how to scrape html generated by javascript using python?

I want to scrape the html generated by javascript , just like what you can see in Firebug.

UPDATE: I've found this article: http://blog.motane.lu/2009/07/07/downloading-a-pages-content-with-python-and-webkit/ which use PyQt to solve the problem and it works well for me.

BUT another problem occur: I have to login the website first, but I don't know how to s开发者_运维知识库imulate login in PyQt .... :(

Have a look at this article which describes using Windmill to do scrape a page after Javascript has been executed by the browser.

This article will show how to extract the desired information using the same three steps when the web page is not written directly using HTML, but is auto-generated using JavaScript to update the DOM tree.

They have some examples I am sure you can easily adapt.

To be precise with terminology, Javascript does not generate HTML. Javascript generates and manipulates the DOM in your browser. The Firebug is showing you HTML representation of that DOM so that it would be readable. The HTML does not actually exist. :)

I don't think an out-of-the box easy solution exists. You may want to look at this blog post and comments that have some pointers.

You could use python spidermonkey, which is a python wrapper to Firefox's engine: http://code.google.com/p/python-spidermonkey/

But the project seems a bit immature.