Trouble Scraping Web Page With Malformed Content
I have written c# code which utilizes the HtmlAgilityPack library in order to scrape a page located at: World's Largest Urban Areas (Page 2). Unfortunately the page consists of malformed content.
I'm at an impasse on how to scrape this page. The current code I have (appearing below) freezes on parsing the HTML:
HtmlNodeCollection cityRecords = _htmlDocument.DocumentNode.SelectNodes("//table[@class='boldtable']//tr[position() != 1]");
CityNodes = (from node in cityRecords.Descendants()
where node.Name == "td"
select node).ToList();
The goal is to parse each and every city listed on the pa开发者_开发技巧ge with each of the data points; nothing more. Looking for recommendations on how to modify the above code or use another freely available library.
Thanks!
Run the content through HTML Tidy before parsing it.
http://tidy.sourceforge.net/
精彩评论