Extracting data from a web page

I am doing a school project which needs extracting data from web pages. To be precise I need a library or opensource program to extract human readable content from html/text data. Something like web browser rendered text content.

I know parsing html with regexs is worst method to extract text from it.

Extra info:

I need it for computing similarity开发者_运维技巧 between text documents.

Any help would be appreciated. Thanks

I would highly recommend this question's first answer in an effort to keep you away from parsing HTML with regular expressions. That answer does a far better job of illustrating why you shouldn't than I could, so I defer to that.

You will also find that you should look into XML parsers instead of trying to "parse by hand" via a regex (which you'll read in the referenced question and its answers).

If all you care is textual similarity, you could just write a regex to strip out all the HTML tags of the form </?(every|single|valid|tag)[^>]*> (perhaps first removing all <script>.*</script> tags), then mash all the content up in a very long paragraph. That wouldn't be a bad use of a regex at all; that's what they're there for.

I might recommend http://docs.python.org/library/xml.dom.minidom.html , but imho the interface can be very awkward. Also you don't need access to the hierarchical structure, just the text. Otherwise a parser would be better than a regex (which would otherwise be a terrible idea).