Django: start a process in a background thread?

I'm trying to work out how to run a process in a background thread in Django. I'm new to both Django and threads, so please bear with me if 开发者_StackOverflowI'm using the terminology wrong.

Here's the code I have. Basically I'd like start_processing to begin as soon as the success function is triggered. However start_processing is the kind of function that could easily take a few minutes or fail (it's dependent on an external service over which I have no control), and I don't want the user to have to wait for it to complete successfully before the view is rendered. ('Success' as far as they are concerned isn't dependent on the result of start_processing; I'm the only person who needs to worry if it fails.)

def success(request, filepath):
    start_processing(filepath)
    return render_to_response('success.html', context_instance = RequestContext(request))

From the Googling I've done, most people suggest that background threads aren't used in Django, and instead a cron job is more suitable. But I would quite like start_processing to begin as soon as the user gets to the success function, rather than waiting until the cron job runs. Is there a way to do this?

If you really need a quick hack, simply start a process using subprocess.

But I would not recommend spawning a process (or even a thread), especially if your web site is public: in case of high load (which could be "natural" or the result of a trivial DoS attack), you would be spawning many processes or threads, which would end up using up all your system resources and killing your server.

I would instead recommend using a job server: I use Celery (with Redis as the backend), it's very simple and works just great. You can check out many other job servers, such as RabbitMQ or Gearman. In your case, a job server might be overkill: you could simply run Redis and use it as a light-weight message server. Here is an example of how to do this.

Cheers

In case someone really wants to run another thread

def background_process():
    import time
    print("process started")
    time.sleep(100)
    print("process finished")

def index(request):
    import threading
    t = threading.Thread(target=background_process, args=(), kwargs={})
    t.setDaemon(True)
    t.start()
    return HttpResponse("main thread content")

This will return response first, then print "process finished" to console. So user will not face any delay.

Using Celery is definitely a better solution. However, installing Celery could be unnecessary for a very small project with a limited server etc.

You may also need to use threads in a big project. Because running Celery in all your servers is not a good idea. Then there won't be a way to run a separate process in each server. You may need threads to handle this case. File system operations might be an example. It's not very likely though and it is still better to use Celery with long running processes.

Use wisely.

I'm not sure you need a thread for that. It sounds like you just want to spawn off a process, so look into the subprocess module.

IIUC, The problem here is that the webserver process might not like extra long-running threads, it might kill/spawn server processes as demand go up and down, etc etc.

You're probably better of by communicating to an external service process for this type of processing, instead of embedding it in in the webserver's wsgi/fastcgi process.

If the only thing you're sending over is the filepath, it ought to be pretty easy to write that service app.