How to save a whole web page with something block in it
I want to save a web page. I use python urllib to parse the web page. But I
find the saved file, where some content is missing. The missing part
is block from the source web page, such as this part <div
style="display: block;" id="GeneInts">...</div>
.
I don't know how to parse a whole page without something block in it. Could you help me
figure it out? 开发者_JAVA技巧Thank you!
This is my program
url = 'http://receptome.stanford.edu/hpmr/SearchDB/getGenePage.asp?Param=4502931&ProtId=1&ProtType=Receptor'
f = urllib.urlretrieve(url,'test.html')
Whenever I need to let Javascript operate on a page before I can scrape it, the first thing I always turn to is SeleniumRC -- while it's mainly designed for purposes of testing, I've never found a better tool for this challenging task. For the "using it from Python" part, see here and links therefrom.
That page generates a great deal of its content with JavaScript executed at load-time, including, I think, the part you're trying to extract. You need a screen-scraper that can run JavaScript and then save out the modified DOM. I don't know where you get one of those.
精彩评论