Persistent delayed jobs queue for Java

I'm looking for an existing system to replace an existing slow and complicated self written mechanism of job management. The existing system:

• 1 MySQL DB with a long massive table of jobs - the queue
• Multiple servers (written in java) all extracting jobs from the queue and processing them
• a job might NOT be deleted from the queue after processing it, to rerun it later
• a job might create other jobs and insert them to the queue

The limitations:

• As more and more jobs are created and inserted in to the queue, it takes longer to extract jobs from it. (The jobs are chosen by priority and type) - create a bottle neck

I'm lo开发者_运维知识库oking for an existing system that can replace this one, and improve it's performance. Any suggestions?

Thanks

I don't generally recommend JMS, but it sounds like it really is what you need here. Distributed, transactional, persistent job queue management is what JMS is all about.

Popular open-source implementations include HornetQ and ActiveMQ.

You could:

• submit your jobs to Amazon's Simple Queue Service (maybe JAXB marshalled)
• dynamically start some EC2 instances according to your queue's length and probably
• submit the results (or availability notice for some files on S3) to Simple Notification Service (again JAXB marshalled).

That exactly what we do, using EC2 Spot instances to minimize costs. And that's what I call serious cloud computing ;)