Delayed_job returns "syntax error" for process daemon and rake task

Started to have an issue this afternoon with the delayed_job process, found that the daemons were dead and restarted them...

Roughly 40 minutes later the process died again @ 86 jobs. Tried to start the daemons again and found they quietly died immediately after I start them. Checking the log I found:

s603021@ec2-184-72-244-232:~/rails/gamestreamer/current$ tail -F log/delayed_job.log 
  SQL (0.7ms)   SET NAMES 'utf8'
  SQL (0.8ms)   SET SQL_AUTO_IS_NULL=0
*** Starting job worker delayed_job.4 host:ec2-184-72-244-232.compute-1.amazonaws.com pid:17397
  Delayed::Job Columns (1.5ms)   SHOW FIELDS FROM `delayed_jobs`
  Product Columns (1.5ms)   SHOW FIELDS FROM `products`
  Game Columns (2.4ms)   SHOW FIELDS FROM `games`
  Delayed::Job Update (12.8ms)   UPDATE `delayed_jobs` SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job.4 host:ec2-184-72-244-232.compute-1.amazonaws.com pid:17397') 
#<ArgumentError: syntax error on line 1588, col 10: `'>

I went ahead and did some digging, a lot of people used to find the daemon gem for delayed_jobs was bugged, and the rake task still worked; this was not the case for me however:

s603021@ec2-184-72-244-232:~/rails/gamestreamer/current$ RAILS_ENV=gs1_development rake jobs:work --trace
(in /data01/home/s603021/rails/gamestreamer/releases/20110302193304)
** Invoke jobs:work (first_time)
** Invoke merb_env (first_time)
** Execute merb_env
** Invoke environment (first_time)
** Execute environment
** Execute jobs:work
*** Starting job worker host:ec2-184-72-244-232.compute-1.amazonaws.com pid:17082
rake aborted!
syntax error on line 1588, col 10: `'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/1.8/yaml.rb:133:in `load'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/1.8/yaml.rb:133:in `load'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:234:in `deserialize'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:50:in `payload_object'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:55:in `name'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:90:in `run_with_lock'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:159:in `reserve_and_run_one_job'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:158:in `each'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:158:in `reserve_and_run_one_job'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:205:in `work_off'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:204:in `times'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/job.rb:204:in `work_off'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/worker.rb:30:in `start'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/data01/home/s603021/rails/games开发者_JAVA百科treamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/worker.rb:29:in `start'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/worker.rb:26:in `loop'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/worker.rb:26:in `start'
/data01/home/s603021/rails/gamestreamer/releases/20110302193304/vendor/plugins/delayed_job/lib/delayed/tasks.rb:13
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31
/usr/bin/rake:19:in `load'
/usr/bin/rake:19

Now I'm simply not sure what could possibly be the problem, any advice on how to get more information(or a more verbose exception) would be awesome. If anyone can help me to solve it, that'd be unbelievable. Thanks in advance--

Job.rb File: http://pastie.org/1697465

I ran into a similar problem a couple of months back where my stack trace spat out:

2011-02-02 20:19:42 INFO -- Processing request 342 for service document_upload, worker 1
2011-02-02 20:19:44 FATAL -- ArgumentError (syntax error on line 29, col -1: `'):
    /usr/lib/ruby/1.8/yaml.rb:133:in `load'
    /usr/lib/ruby/1.8/yaml.rb:133:in `load'
    /var/local/hprime/lib/puck_seed_file.rb:179:in `get_seed'
    /var/local/hprime/lib/puck_seed_file.rb:7:in `read'

Took me ages to find what was causing it because it was pretty intermittent, but turned out to be a problem with the syntax in the actual job being processed. Not sure exactly what yours is doing but you could start there..?

Also while trying to fix my prob I came across a few people who had fixed similar problems by changing the parsing engine in yaml by adding this to their config/boot.rb

require 'yaml'
YAML::ENGINE.yamler= 'syck'

Apparently the default parser spits the dummy if special characters are included in the yaml content.. You could try that also and see if it helps. Anyways, sorry it's a bit vague, hope it helps.

The issue came down a job payload that YAML was unable to parse -- cred+ to 2potatocakes for the closest answer.

Loading up IRb and checking jobs against the payload_object method will show you which job YAML is choking on, destroy the job and the rake task will work fine.