What's the best way to 'ping' thousands of servers every minute?

I run a server monitoring site for a video game. It monitors thousands of servers (currently 15,000 or so).

My current setup is a bit janky, and I want to improve it. Currently I use cron to submit every server to a resque job queue. I refill the queue just as soon as it's empty, essentially creating a constantly working queue. The job will then simply try and open a socket connection to the server ip and port in question, and mark it down if it fails to connect.

I have 20 workers, and it gets the job done in about 5 minutes. I feel that this should be able to go MUCH faster.

Is there a bett开发者_开发技巧er, quicker way of doing this?

So, what you are doing currently I assume is doing a TCP socket connection which pings your game server. The problem with using TCP is obviously that it is a lot slower than UDP. What I would advise instead is creating a UDP socket that just checks for the game server port.

Here's a nice quote from another question:

>        UDP is really faster than TCP, and the simple reason is because
> it's non-existent acknowledge packet (ACK) that permits a continuous
> packet stream, instead of TCP that acknowledges each packet.

Read this question here: UDP vs TCP, how much faster is it?

From my experience with game servers, the majority if not 100% of all modern game servers allow you to query them on a UDP socket. This will then respond with details on the game server. (I used to host a lot of servers myself too).

So basically, make sure that you are using UDP rather than TCP...

Example Query

I'm just searching for this information now and will update my question...when I find some source.. what game is it that you are trying to get information for?

Use typical solutions for typical tasks. This case is about available detection every n seconds - one of daily sysadmin task. It should not be over ICMP, use SNMP over UDP proto. One of complete solution is Nagious/Cacti/Zabbix, which have built-in functionality to combine everything about your servers: LA, HDD, RAM, IO, NET as well as available detection.

You don't mention how you are making the socket connections, but you might want to try using ruby curl bindings: curb instead of net/http.

This will typically be much faster.