Which Screen Scraper should I use for Dynamic Webpages?

I m looking to develop a program, which collects data from 6 different websites and displaying information which changes dynamically.

In order to develop such program I require a screen scraper utility which is able to extract data simultaneously from 6 different WebPages on which the data changes at a very high frequency.

See the link below to get an idea of the type and the dynamic manner in which the data is displa开发者_JAVA百科yed.

http://www.igmarkets.com.au/pricestream/outside/getMarketList?webSiteId=zam&businessArea=F&locale=en_ZA&firstTime=true&skin=igindex

The info is displayed somewhat differently on the 6 webpages, but the concept is very similar.

Can anyone please advise what is the correct utility software which is ideal for such purpose.

Many thanks

Ethically screen scraping can be a dubious practice - but there are legitimate uses. Search Engines do a kind of scrape to get their results, as an example.

Some sites will block utilities that make too many requests - which is a good reason to ask permission if you are going to make many requests - as it may otherwise look like a DoS attack and the site may take action against you.

All that aside if you have a legitimate use to read the contents of a page (so that you can process it in some way) the code is fairly trivial.

From MSDN

In C#:

// Initialize the WebRequest.
WebRequest myRequest = WebRequest.Create("http://www.contoso.com");

// Return the response. 
WebResponse myResponse = myRequest.GetResponse();

// Code to use the WebResponse goes here.

// Close the response to free resources.
myResponse.Close();