Asynchronous task system on top of Rails?

I'm currently managing a legacy rails ap开发者_开发技巧plication that's running on rails 1.2.7. One of the functionalities is allowing people to upload sounds and have them converted using a command line tool using backticks. Currently I use an AJAX poll to do the management of the conversion through a controller action, but I'm having issues with timeouts, meaning that the final elements of the controller action simply are not occurring.

It's a system that requires low overheads, what could I use to manage this background conversion and then respond in an evented system to issues created by the background conversion? I was looking at eventmachine but I'm still not 100% with it, are there any other kind of asynchronous task based systems I could use?

First of all, wow, Rails 1.2.7. I have a similarly aged app at work which I'm slowly upgrading to Rails 3. Crazy how fast this stuff changes.

Definitely a fun problem. There are lots of directions you could take this, and I'm not sure which is best, as I'm not sure I understand your process. So I'll suggest a couple. My understanding is 1) Upload file, 2) Start conversion, 3) Report on conversion status via ajax polling.

First, running the conversion utility in a Rails controller action is definitely not the way to go, as you've discovered. 1) Your Web server or browser will probably kill the request, and 2) most Rails deployments allow for only 1 request at a time per app, meaning if you want 5 simultaneous users uploading, you need 5 copies of your app running. Obviously that won't scale.

Your "upload" action should be as quick as possible. It should 1) upload the file, and 2) either schedule or fire off a "conversion job", which some other process would handle. Your polling action would then just report on the status of that job. Of course the question is what that other process should be.

Idea 1 http://geekblog.vodpod.com/2007/08/17/background-processing-in-rails/ is likely a good place to start, though I cannot vouch for that approach.

Idea 2 I've done something similar to this, so I can give more detail. And it probably scales better. Build a light-weight companion app using Sinatra or Async Sinatra. Your Rails app would record a job for the uploaded file in your database, but then its part is done. Your Sinatra app, using EventMachine, would poll the db every few seconds and start new jobs. You might want to limit it to n concurrent jobs so you don't DOS your own box :) Your users could then poll your Sinatra app to get their conversion's status.

Idea 3 Similar to 2, but instead of a companion Web app, it's just a small Ruby program using EventMachine. You would just start this program on your server and let it run forever. Each job would write its status back to the database, which your users could poll through your Rails app. I think this is my favorite. Wireframe:

#!/usr/bin/env ruby
require 'rubygems'
require 'eventmachine'

# Returns new jobs from the database
def new_jobs
  []
end

# Convert the file
def convert(job)
  `convert #{job.path}`
  job
end

# Callback when conversion is complete
def callback(job)
  puts "Finished #{job.path}!"
end

EventMachine::run do
  # Run every 5 seconds
  EventMachine::add_periodic_timer(5) do
    new_jobs.each { |job| EventMachine::defer convert(job), callback(job) }
  end
end

These suggestions are admittedly from a 10,000 ft. view, but I hope there's something in there to get you started.