开发者

How to save a whole web page with something block in it

I want to save a web page. I use python urllib to parse the web page. But I find the saved file, where some content is missing. The missing part is block from the source web page, such as this part <div style="display: block;" id="GeneInts">...</div>. I don't know how to parse a whole page without something block in it. Could you help me figure it out? 开发者_JAVA技巧Thank you!

This is my program

url = 'http://receptome.stanford.edu/hpmr/SearchDB/getGenePage.asp?Param=4502931&ProtId=1&ProtType=Receptor'
f = urllib.urlretrieve(url,'test.html') 


Whenever I need to let Javascript operate on a page before I can scrape it, the first thing I always turn to is SeleniumRC -- while it's mainly designed for purposes of testing, I've never found a better tool for this challenging task. For the "using it from Python" part, see here and links therefrom.


That page generates a great deal of its content with JavaScript executed at load-time, including, I think, the part you're trying to extract. You need a screen-scraper that can run JavaScript and then save out the modified DOM. I don't know where you get one of those.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜