开发者

Jruby/Resque: Network connections from resque job begin failing repeatedly

I've having a bizarre issue with a resque job that I'm wondering if anyone else has run into.

We're running resque under jruby 1.6.2

We have a long running task that downloads a bunch of files from various URLs, uploads those files to Rackspace Cloudfiles using Fog, and then stores some information about those files in MySQL. After this has been going on for awhile, it seems like our application's networking stack falls over. In one instance, the first sign of failure was a timeout from here:

org/jruby/ext/openssl/SSLSocket.java:512:in `sysread'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/jruby-openssl-0.7.4/lib/openssl/buffering.rb:35:in `fill_rbuff'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/jruby-openssl-0.7.4/lib/openssl/buffering.rb:158:in `eof?'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/jruby-openssl-0.7.4/lib/openssl/buffering.rb:133:in `readline'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/excon-0.6.5/lib/excon/response.rb:22:in `parse'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/excon-0.6.5/lib/excon/connection.rb:174:in `request'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/fog-0.11.0/lib/fog/core/connection.rb:20:in `request'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/fog-0.11.0/lib/fog/storage/rackspace.rb:107:in `request'

We usually start seeing these problems appear about 10-15 minutes into the job run. After that, we start seeing this on every subsequent attempt to 开发者_如何学Cwrite to the database...

ActiveRecord::JDBCError: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.: SELECT `bills`.* FROM `bills` WHERE (`bills`.state_session_id = 59)
ActiveRecord::StatementInvalid: ActiveRecord::JDBCError: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.: SELECT `bills`.* FROM `bills` WHERE (`bills`.state_session_id = 59)
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract_adapter.rb:207:in `log'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract_adapter.rb:200:in `log'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-jdbc-adapter-1.1.1/lib/arjdbc/jdbc/adapter.rb:183:in `execute'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-jdbc-adapter-1.1.1/lib/arjdbc/jdbc/adapter.rb:275:in `select'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:56:in `select_all'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/base.rb:473:in `find_by_sql'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/relation.rb:64:in `to_a'
/var/www/lisausa/shared/bundle/jruby/1.9/gems/activerecord-3.0.10/lib/active_record/relation/finder_methods.rb:143:in `all'

I've tried using the ruby-cloudfiles gem instead of Fog, but we seem to start running into the exact same errors eventually using that combination too. If I disable the file download/cloudfiles upload piece of this, these errors never appear, and I've been able to leave this particular job running for a number of days.

Any theories on what could be happening here?


For what it's worth, months later, I can tell you that

"ActiveRecord::JDBCError: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost..."

has typically been an indicator for us that the database connection has been broken (either because of a network problem, or the DB went down, or etc.) and the underlying socket closed. For us at least, JRuby/ActiveRecord/ConnectionPool's default behavior is to keep blowing up with this error on every command instead of reconnecting.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜