Passenger concurrent connections error

I have production server (Nginx + Passenger). When I trying to ab -n 3 -c 3 myhost.ru/ from another computer I get this error log in my nginx error.log:

[ pid=21160 thr=139775297914624 file=ext/nginx/HelperAgent.cpp:584 time=2011-08-31 15:25:49.22 ]: Uncaught exception in PassengerServer client thread:
   exception: Cannot read response from backend process: Connection reset by peer (104)
   backtrace:
     in 'void Client::forwardResponse(Passenger::SessionPtr&, Passenger::FileDescriptor&, const Passenger::AnalyticsLogPtr&)' (HelperAgent.cpp:330)
     in 'void Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:510)
     in 'void Client::threadMain()' (HelperAgent.cpp:603)

[ pid=21249 thr=15884760 file=utils.rb:176 time=2011-08-31 15:25:49.022 ]: *** Exception PGError in application (server closed the connection unexpectedly
  This probably means the server terminated abnormally
  before or while processing the request.
) (process 21249, thread #<Thread:0x00000001e4c3b0>):
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:276:in `exec'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:276:in `block in clear_cache!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:275:in `each_value'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:275:in `clear_cache!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:303:in `disconnect!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:202:in `block in disconnect!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:201:in `each'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:201:in `disconnect!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/core_ext/module/synchronization.rb:35:in `block in disconnect_with_synchronization!'
  from /home/deployer/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/core_ext/module/synchronization.rb:34:in `disconnect_with_synchronization!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `block in clear_all_connections!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `each_value'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `clear_all_connections!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_specification.rb:123:in `clear_all_connections!'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:398:in `before_handling_requests'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:204:in `start_request_handler'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:170:in `block in handle_spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:479:in `safe_fork'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:165:in `handle_spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:180:in `start'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_sp开发者_Python百科awner.rb:128:in `start'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
  from <internal:prelude>:10:in `synchronize'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/helper-scripts/passenger-spawn-server:99:in `<main>'
2011/08/31 15:25:49 [error] 21197#0: *41173 upstream prematurely closed connection while reading response header from upstream, client: 78.47.253.164, server: yandex.ru, request: "GET / HTTP/1.0", upstream: "passenger:unix:/passenger_helper_server:", host: "yandex.ru"
[ pid=21160 thr=139775298115328 file=ext/nginx/HelperAgent.cpp:584 time=2011-08-31 15:25:49.31 ]: Uncaught exception in PassengerServer client thread:
   exception: Cannot read response from backend process: Connection reset by peer (104)
   backtrace:
     in 'void Client::forwardResponse(Passenger::SessionPtr&, Passenger::FileDescriptor&, const Passenger::AnalyticsLogPtr&)' (HelperAgent.cpp:330)
     in 'void Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:510)
     in 'void Client::threadMain()' (HelperAgent.cpp:603)

2011/08/31 15:25:49 [error] 21197#0: *41172 upstream prematurely closed connection while reading response header from upstream, client: 78.47.253.164, server: yandex.ru, request: "GET / HTTP/1.0", upstream: "passenger:unix:/passenger_helper_server:", host: "yandex.ru"
[ pid=21257 thr=15884760 file=utils.rb:176 time=2011-08-31 15:25:49.032 ]: *** Exception PGError in application (server closed the connection unexpectedly
  This probably means the server terminated abnormally
  before or while processing the request.
) (process 21257, thread #<Thread:0x00000001e4c3b0>):
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:276:in `exec'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:276:in `block in clear_cache!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:275:in `each_value'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:275:in `clear_cache!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/postgresql_adapter.rb:303:in `disconnect!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:202:in `block in disconnect!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:201:in `each'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:201:in `disconnect!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/core_ext/module/synchronization.rb:35:in `block in disconnect_with_synchronization!'
  from /home/deployer/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/core_ext/module/synchronization.rb:34:in `disconnect_with_synchronization!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `block in clear_all_connections!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `each_value'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `clear_all_connections!'
  from /home/deployer/yandex.ru/shared/bundle/ruby/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_specification.rb:123:in `clear_all_connections!'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:398:in `before_handling_requests'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:204:in `start_request_handler'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:170:in `block in handle_spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:479:in `safe_fork'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:165:in `handle_spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:180:in `start'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:128:in `start'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
  from <internal:prelude>:10:in `synchronize'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  from /home/deployer/.rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/helper-scripts/passenger-spawn-server:99:in `<main>'

I found solution myself. There is passenger_spawn_method conservative must be written in nginx.conf instead of default passenger_spawn_method conservative smart-lv2.

Are you allocating or managing your own database connections/pools?

Passenger re-connects the default ActiveRecord database pool for each background worker after it forks.

However, other resources or manually managed database connections need to be manually setup/reconnected for each worker after forking.

I wrote up a detailed blog post explaining how to fix this: http://blog.winfieldpeterson.com/2011/01/06/passenger-resource-collision/

This sounds like what you're experiencing, multiple workers trying to share a single database connection and producing errors under load due to resource contention.

We had the same issue with Rails 3.1, pg, ruby 1.9.3, and using Apache + Passenger 3.0.9. Upgrading to Rails 3.1.1 seems to resolve this issue.

https://github.com/rails/rails/issues/1339