Improving ruby code performance with threading not working

I am trying to optimize this piece of ruby code with Thread, which involves a fair bit of IO and network 开发者_如何学Pythonactivity. Unfortunately that is not going down too well.

# each host is somewhere in the local network
hosts.each { |host|
  # reach out to every host with something for it to do
  # wait for host to complete the work and get back
}

My original plan was to wrap the internal of the loop into a new Thread for each iteration. Something like:

# each host is somewhere in the local network
hosts.each { |host|
  Thread.new { 
  # reach out to every host with something for it to do
  # wait for host to complete the work and get back
  }
}

# join all threads here before main ends

I was hoping that since this I/O bound even without ruby 1.9 I should be able to gain something but nope, nothing. Any ideas how this might be improved?

I'm not sure how many hosts you have, but if it's a lot you may want to try a producer/consumer model with a fixed number of threads:

require 'thread'

THREADS_COUNT = (ARGV[0] || 4).to_i

$q = Queue.new
hosts.each { |h| $q << h }

threads = (1..THREADS_COUNT).map {
  Thread.new {
    begin
      loop {
        host = $q.shift(true)

       # do something with host
      }
    rescue ThreadError
      # queue is empty, pass
    end
  }
}

threads.each(&:join)

If this all too complicated, you may try using xargs -P. ;)

@Fanatic23, before you draw any conclusions, instrument your code with puts and see whether the network requests are actually overlapping. In each call to puts, print a status string indicating the line which is executing, along with Time.now and Time.now.nsec.

You say "since this [is] I/O bound even without ruby 1.9 I should be able to gain something".

Vain hope. :-)

When a thread in Ruby 1.8 blocks on IO, the entire process has blocked on IO. This is because it's using green theads.

Upgrade to Ruby 1.9, and you'll have access to your platform's native threads implementation. For more, see:

http://en.wikipedia.org/wiki/Green_threads

Enjoy!