resque worker on multiple servers

Because of heavy background process, we need multiple servers that just do background processes. We are using resque for handling background jobs. Since the res开发者_开发百科que workers on each server will be doing the similar tasks, what would be the best way to do this?

1. Start workers on each machine, each processing all the queues.
2. Divide up the queues between the workers on each servers so that workers on one server will be responsible for handling half of the queues while workers on next for the rest of the queues.

I like the first solution. But is there any limitation with this?

Thanks.

You should be able to spread your Resque workers across any number of servers without any problem.

Resque work queues should be used to categorize and organize different types of work.

I'm personally of the opinion every worker should be able to process every type of job, so you get an efficient use of resources.

That being said, I change the priority order of what queues get processed on different workers so that a given type of job is likely to find a worker to help even if most of the resources are dedicated to higher priority work.

For example, with 6 workers and 3 types of queus:

Queues:
-------
critical - top priority jobs
email - time-sensitive, but not critical
maintenance - background work, to be done whenever

Workers w/ Queues
-----------------
1: critical, email, maintenance
2: critical, email, maintenance
3: critical, email, maintenance
4: critical, email, maintenance
5: critical, maintenance, email
6: critical, maintenance, email

This setup guarentees critical work will be processed before any other work on any worker. However, if the system is inundated with email jobs, only 2/3 of the workers will be blocked if there are maintenance jobs that need to happen.

Hopefully you can extrapolate this example to your queues and priorities.