Scaling cronjobs over multiple servers

right now, we have a single server with a cronjob tab that sends out daily emails. We would like to scale that server. The application is standard z开发者_开发百科end framework application deployed on centos server in amazon cloud.

We already took care of the load balancing, content management and managing deployment. However, the cronjob is still an issue for us, as we need to grantee that some jobs are performed only once.

For example, the daily emails cronjob must only be executed once by a single server. I'm looking for the best method to grantee only one server will execute it only once.

I'm thinking about 2 solutions, but i was wondering if someone else had the same issue.

1. Make one of the servers "master", who only sends out the daily emails. That will be an issue, if the server malfunction, and generally we don't want to have a "special" server. It would also means we will need to keep track which server is master.
2. Have a queue of schedule tasks to be performed. Each server open that queue and sees which tasks needed to be performed. The first server who "grab" the task, will preform the task and mark it as done. I was looking at amazon simple queuing service as a solution for the queue.

Both these solutions have advantages and disadvantages, and i was wondering if someone thought about someone else that might help us here.

When you need to scale out cron jobs, you are better off using a job manager like Gearman

Beanstalkd could also be an option for you.

I had the same problem. What I did was dead simple.

1. I spun up the cheapest EC2 instance on AWS.
2. I created the cronjob(s) only on this server.
3. The cron job just run jobs that only makes a simple request to my endpoint / api (i.e. api.mydomain.com).
4. On my api, i just have a route watching for these special request that will run the job I want. So basically, all I'm doing instead of running the task using a cronjob, im running the task via a http request.

I hope that makes sense! Now it doesn't matter how many servers you have, it will just scale! Also, your cronjob server's only function is to run dead simple jobs to send a request, nothing more.