Python- is there a module that will automatically scrape the content of an article off a webpage?

I know there is lxml and BeautifulSoup, but that won't work for my project, because I don't know in advance what the H开发者_如何转开发TML format of the site I am trying to scrape an article off of will be. Is there a python-type module similar to Readability that does a pretty good job at finding the content of an article and returning it?

It's possible to do using PhantomJS (C++) or PyPhantomJS (Python).

They're both headless WebKit based browsers, which you can fully control from JavaScript. Because you can control it from JavaScript, I find it is really easy to do stuff such as scrape the content of an article.

PyPhantomJS also has a plugin system, so that's definitely a plus. :)

Extracting the real content from a content-page can not be done automatically - at least not with the standard tools. You have to define/identify where the real content is stored (by specifying the related CSS ID or class in your own HTML extraction code).

Using HTQL, the query is: &html_main_text