Making only one task run at a time in celerybeat

I have a task which I execute once a minute using celerybeat. It works fine. Sometimes though, the task takes a few seconds more than a minute to run because of which two instances of the task run. This leads to some race conditions that mess things up.

I can (and probably should) fix my task to work properly but I wanted to know if celery has any builtin ways to ensur开发者_StackOverflowe this. My cursory Google searches and RTFMs yielded no results.

You could add a lock, using something like memcached or just your db.

If you are using a cron schedule or time interval for run periodic tasks you will still have the problem. You can always use a lock mechanism using a db or cache or even filesystem or also schedule the next task from the previous one, maybe not the best approach. This question can probably help you: django celery: how to set task to run at specific interval programmatically

You can try adding a classfield to the object that holds the function that youre making run and use that field as a "some other guy is working or not" control

The lock is a good way with either beat or a cron.

But, be aware that beat jobs run at worker start time, not at beat run time.

This was causing me to get a race condition even with a lock. Lets say the worker is off and beat throws 10 jobs into the queue. When celery starts up with 4 processes, all 4 of them grab a task and in my case 1 or 2 would get and set the lock at the same time.

Solution one is to use a cron with a lock, as a cron will execute at that time, not at worker start time.

Solution two is to use a slightly more advanced locking mechanism that handles race conditions. For redis look into setnx, or the newer redlock. This blog post is really good, and includes a decorator pattern that uses redis-py's locking mechanism: http://loose-bits.com/2010/10/distributed-task-locking-in-celery.html.