Ruby: How to screen-scrape the result of an Ajax request

I have written a ruby script to screen scrape something using the 'open-uri' and 'hpricot' gems - everything works great so far.

But now I have to screen scrape something which is returned after a form is submitted via a javascript function (called by an 'onchange' event handler from a drop-down menu):

function submit_form() {
  document.list.action="/some/sort/of/path";
  document.list.submit();
}

AFAIK, open-uri lets you submit only GET requests. And if I'm not mistaken, a POST request would be needed here.

So my question is: what do I need to install and to 'require' and how would the ruby code then look like (to m开发者_运维百科ake that POST request) - sorry, I'm still pretty much of a n00b...

Thank you very much for your help!

Tom

I think you definitely should use Mechanize. It provides a nifty interface to interact with remote pages, forms on them, and so forth (see this example).

The Ruby standard library has the http class, which naturally supports the POST operation.

Net::HTTP.post_form(URI.parse('http://www.example.com/some/sort/of/path')

If you find the API there less than optimal, then take a look at the httparty gem

Finally, while hpricot is a great gem, it isn't actively developed any longer. You should consider moving to nokogiri which practically replaces hpricot and improves upon it.