Uploading lots of files in my webapp leads to deadlock in Rack + Unicorn

I've got a Rails 3.1rc1 app that allows uploading of multiple files at once via a jQuery plugin called File Upload: http://aquantum-demo.appspot.com/file-upload This allows the user to upload a bunch of images which I then resize by farming out to ImageMagick's convert command and upload to S3. My site is served by two Unicorn workers.

Last night I was uploading a batch of about 30 images at once. It looks like Chrome will do 6 simultaneously and it got through about half the batch before this started showing up in my unicorn.stderr.log:

E, [2011-06-01T07:01:39.806164 #21751] ERROR -- : Read error: #<ThreadError: deadlock; recursive locking>
E, [2011-06-01T07:01:39.806316 #21751] ERROR -- : /var/www/shared/bundle/ruby/1.9.1/gems/rack-1.3.0/lib/rack/lock.rb:33:in `lock'
/var/www/shared/bundle/ruby/1.9.1/gems/rack-1.3.0/lib/rack/lock.rb:33:in `call'
/var/www/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:132:in `forward'
/var/www/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:243:in `fetch'
/var/www/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:181:in `lookup'
/var/www/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:65:in `call!'
/var/www/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:50:in `call'
/var/www/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc1/lib/rails/rack/content_length.rb:16:in `call'
/var/www/shared/bundle/ruby/1.9.1/gems/hoptoad_notifier-2.4.10/lib/hoptoad_notifier/user_informer.rb:12:in `call'
/var/www/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc1/lib/rails/engine.rb:438:in `call'
/var/www/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc1/lib/rails/railtie/configurable.rb:28:in `method_missing'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:545:in `process_client'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:623:in `block in worker_loop'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:621:in `each'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:621:in `worker_loop'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:509:in `block (2 levels) in spawn_missing_workers'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:507:in `fork'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:507:in `block in spawn_missing_workers'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:503:in `each'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:503:in `spawn_missing_workers'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:516:in `maintain_worker_count'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn/http_server.rb:166:in `start'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/lib/unicorn.rb:30:in `run'
/var/www/shared/bundle/ruby/1.9.1/gems/unicorn-3.6.2/bin/unicorn:120:in `<top (required)>'
/var/www/shared/bundle/ruby/1.9.1/bin/unicorn:19:in `load'
/var/www/shared/bundle/ruby/1.9.1/bin/unicorn:19:in `<main>'

The rest of my image uploads timed out and Unicorn restarted itself. I could upload again, but doing multiple开发者_开发百科s at once kept causing this error (and much more frequently after the first time it happened--I could sometimes get it to deadlock when uploading two files at the same time).

I'm hoping for some ideas as to where to start troubleshooting. What part of the request is being deadlocked? None of my Rails code shows anywhere in that stack so I assume it's happening in the hardcore request logic which makes me think it won't be an easy fix. Should I fall back to something less event-driven like Mongrel for my file uploads? I assume that means you'll only be able to upload as many files simultaneously as there are instances of mongrel running, but at least the whole site won't go down if someone uploads a huge batch of files (which I'm hoping will be a common occurrence).

Thanks for any help!

Update I'm seeing the ImageMagick convert command locking quite a bit...I'm not sure how the internals of Rack works when you've got a request that's "stuck" on the server. I'm going to switch to freeimage and see what happens. I'm also going to take a look at Rainbows! as an alternative to Unicorn on the upload page as these are long running requests which Unicorn admits it's not good at.

You need to configure the jQuery upload plugin to upload one file at a time (it is possible, as far as I remeber with that plugin). It's not Google Chrome's problem.

Also use some kind of background job to process images (eg: delayed_job or resque)

As final note, if you plan that your site will get tons of uploads, maybe you should switch to some external service for multimedia processing (for example, we are using transloadit.com for a project I'm working on).

UPDATE: I found this bug about rack: https://github.com/rack/rack/issues/87

also found your answer on github ;) https://github.com/rack/rack/issues/183

did you solved it?