Bulk API calls in Rails

I have two different Rails application speaking with each other through API calls. Gone were the days where the call was light, meaning there was much less data transfer. We never cared about the HTTP request/response timeout issues. But once the data grew, there we开发者_StackOverflowre timeouts and we had to split data and send it in Batches as opposed to a direct API call.

Currently, we are doing what is needed using Threading and Background job which underneath I feel is not the cleaner way and I'd have to say that performance has taken a huge hit. I feel like I don't have many options. Any suggestions?

Without knowing your specific scenario, I can only suggest the obvious:

Use a queueing system like Resque, requeuing jobs that have timed out automatically and increase the performance of your source (buy hardware, cache stuff.)

If you are for example syncing a model through ActiveResource and you're looping through every single object, resulting in a show call for that object on the source, you should simply pull all objects in question using the index action and loop through those on the receiver rather than requesting one item at a time.