How to parse javascript generated urls in Ruby?

My first question here so let's be brief.

I am parsing this site: http://78.133.214.226/fotoweb/Grid.fwx with ruby mechanize. I get page and post some search term into the form.

I get for example: http://78.133.214.226/fotoweb/Grid.fwx?archiveId=5002&SF_LASTSEARCH=Doda+and+Doda&SF_FIELD1_GROUP=1&SF_GROUP1_BOOLEAN=and&SF_FIELD1_MATCHTYPE=all&SF_FIELD1=Do开发者_运维知识库da&SF_SEARCHINRESULT=0&SF_GROUP2_BOOLEAN=and&SF_GROUP2_FIELD=FQYFT&SF_FIELD2_GROUP=2&SF_FIELD2_MATCHTYPE=exact&SF_FIELD2_BOOLEAN=and&SF_FIELD2=&SF_FIELD3_MATCHTYPE=exact&SF_FIELD3_BOOLEAN=and&SF_FIELD3_GROUP=1&SF_FIELD3=&doSearch=Go

Then I want to parse results to get:

1.description 2.filename 3.url to thumbnail

1 and 2 are easy but I can't get 3. When I inspect image then I can see src of the thumbnail but when I get this element with mechanize there is none.

Any clue were can I look for the right tool for the job?

If you inspect a page in a browser then it is very likely that its DOM has been manipulated by javascript, and so is different than it would appear to Mechanize. Mechanize does not process javascript and so can only get the raw initial html that the website sends to the user. I recommend to use a tool like cURL to get raw html (like Mechanize would see it) and then inspect this version in a browser to decide what you want to pick out later using Mechanize.

I tried curling your page and you are right that there is no src associated with the img tags. Probably done to avoid scraping! You can examine the accompanying javascript and see if there is any relation that you can use to figure out source urls from the data you are able to retrieve.

You can save pages with generated js with phantomjs, then you can parse content from saved files with mechanize