quartz throws deadlocks under load

We are using Quartz with Spring and our configuration is throwing deadlocks when quartz has more than 1 thread configured. I'm starting to believe that it's because we don't have our quartz configured correctly with Spring, but I can't find enough documentation on how to configure the two to play nicely.

We are running on both Windows and Linux environments - pointing at MSSQL and Oracle DBs. With both OS, using either DB, we can throw the following deadlock errors...

We're consistently throwing these deadlock errors. We run under heavy load, inserting hundreds of quartz triggers in a matter of minutes.

2010-03-17 18:52:31,737 [] [] ERROR [DFScheduler_Worker-42] core.ErrorLogger core.ErrorLogger (QuartzScheduler.java:2185) - An error occured while marking executed job complete. job= 'BPM.6e41a6567f0000020100362a51dc7a50'

org.quartz.JobPersistenceException: Couldn't remove trigger: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. [See nested exception: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1469)at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2978)at org.quartz.impl.jdbcjobstore.JobStoreSupport$39.execute(JobStoreSupport.java:2962) at org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3713)at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3747)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3709)at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2958)at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1727)at org.quartz.core.JobRunShell.run(JobRunShell.java:273)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534)

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(Unknown Source at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown Source) at 
...
org.quartz.impl.jdbcjobstore.StdJDBCDelegate.deleteSimpleTrigger(StdJDBCDelegate.java:1820) at org.quartz.impl.jdbcjobstore.JobStoreSupport.deleteTriggerAndChildren(JobStoreSupport.java:1345 at org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1453 ... 9 more

This is what my quartz.properties file looks like:

org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 50
org.quartz.threadPool.threadPriority = 5
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true

org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.开发者_运维问答tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = false
org.quartz.jobStore.selectWithLockSQL = SELECT * FROM {0}LOCKS UPDLOCK WHERE LOCK_NAME = ?

Do you use a SchedulerFactoryBean? The documentation is: http://static.springsource.org/spring/docs/2.5.x/reference/scheduling.html It would completely remove the need for the actual quartz.properties and you can use the datasource you use in the rest of the application. This has obvious advantages (like reusable connections in a connection pool) and at least for me offered a somewhat easier time debugging.

Here's a snippet of how mine looked like in an older project:

<bean id="schedulerFactory" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="quartzProperties"> <!--  quartz attributes, configurable -->
        <value>
            org.quartz.scheduler.rmi.export = false
            org.quartz.scheduler.rmi.proxy = false
            org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
            org.quartz.threadPool.threadCount = ${threadCount}
            org.quartz.threadPool.threadPriority = 5
            org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true
            org.quartz.jobStore.useProperties = false
            org.quartz.jobStore.driverDelegateClass = ${driverDelegate}
            org.quartz.jobStore.tablePrefix = QRTZ_
            org.quartz.jobStore.isClustered = false
        </value>
    </property>
    <property name="autoStartup" value="false" /> 
    <property name="waitForJobsToCompleteOnShutdown" value="false" />
    <property name="dataSource" ref="dataSource" />
</bean>

and the datasource the usual:

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
    ....
    <property name="validationQuery" value="SELECT 1" />
    <property name="testWhileIdle" value="true" />
    <property name="testOnBorrow" value="true" />
    <property name="testOnReturn" value="true" />
</bean>

Obviously the quartzProperties are similar to the ones in the file, but I've found the general behavior of the application better once I switched from the quarts.properties to the Spring classes. It also helped that I had only one datasource actually accessing the database.