ActiveResource EOFError on "slow" API

I'm seriously struggling to solve this one, any help would be appreciated!

I have two Rails apps, let's call them Client and Service, all very simple, normal REST interface - here's the basic scenario:

• Client makes a POST /resources.json request to the Service
• The Service runs a process which creates the resource and returns an ID to the Client

Again, all very simple, just that Service processing is very time-intensive and can take several minutes. If that happens, an EOFError is raised on the Client, exactly 60s after the request was made (no matter what the ActiveResource::Base.timeout is set to) while the service correctly processed the request and responds with 200/201. This is what we see in the logs (chronologically):

C 00:00:00: POST /resources.json
S 00:00:00: Received POST /resources.json => resources#create
C 00:01:00: EOFError: end of file reached
  /usr/ruby1.8.7/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
  /usr/ruby1.8.7/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
  /usr/ruby1.8.7/lib/ruby/1.8/timeout.rb:62:in `timeout'
  ...
S 00:02:23: Response POST /resources.json, 201, after 143s

Obviously the service response never reached the client. I traced the error down to the socket level and recreated the scenario in a script, where I open a TCPSocket and try to retrieve data. Since I don't request anything, I shouldn't get anything back and my request should time out afte开发者_运维知识库r 70 seconds (see full script at the bottom):

Timeout::timeout(70) { TCPSocket.open(domain, 80).sysread(16384) }

These were the results for a few domain:

www.amazon.com     => Timeout after 70s
github.com         => EOFError after 60s
www.nytimes.com    => Timeout after 70s
www.mozilla.org    => EOFError after 13s
www.googlelabs.com => Timeout after 70s
maps.google.com    => Timeout after 70s

As you can see, some servers allowed us to "wait" for the full 70 seconds, while others terminated our connection, raising EOFErrors. When we did this test against our service, we (expectedly) got an EOFError after 60 seconds.

Does anyone know why this happens? Is there any way to prevent these or extend the server-side time-out? Since our service continues "working", even after the socket was closed, I assume it must be terminated on the proxy-level?

Every hint would be greatly appreciated!

PS: The full script:

require 'socket'
require 'benchmark'
require 'timeout'

def test_socket(domain)
  puts "Connecting to #{domain}"
  message = nil
  time    = Benchmark.realtime do
    begin
      Timeout::timeout(70) { TCPSocket.open(domain, 80).sysread(16384) }
      message = "Successfully received data" # Should never happen
    rescue => e
      message = "Server terminated connection: #{e.class} #{e.message}"
    rescue Timeout::Error
      message = "Controlled client-side timeout"
    end
  end
  puts "  #{message} after #{time.round}s"
end

test_socket 'www.amazon.com'
test_socket 'github.com'
test_socket 'www.nytimes.com'
test_socket 'www.mozilla.org'
test_socket 'www.googlelabs.com'
test_socket 'maps.google.com'

I know this is nearly a year old, but in case anyone else finds this, I wanted to add a possible culprit.

Amazon's ELB will terminate idle connections at 60 seconds, so if you are using EC2 behind ELB, then ELB could be the server side problem.

• the only "documentation" I could find here is https://forums.aws.amazon.com/thread.jspa?threadID=33427&start=50&tstart=50, but it's better than nothing

Each server decides when to close the connection. It depends on the server side software and its settings. You can't control that.