Stripping HTML but retaining block/inline structure

I would like to convert HTML to plain text but retain the minimum structure.

• All sections which contain stuff only the browser needs to see such as <script> and <style> to be stripped completely.
• Convert all block tags to <div> and all inline ones to <span> or remove inlines completely without leaving whitespace and turning anything delineatd by block levels into paragraphs with two linebreaks.

The idea is to turn random web pages into something suitable for natural language text processing without artefacts left from naively removing markup artifically break words up or making unrelated blocks look like sentences.

Any binary开发者_开发知识库, library, or source in any programming language is OK.

Is there a standard source preferably machine-readable with a full list of elements defining which are block, which inline, and which are like <script> and <style> above?

The list of HTML 4 Block-level elements is here: http://htmlhelp.com/reference/html40/block.html

The most popular HTML parsing libraries for Perl are HTML::Parser which is a SAX-style parser and HTML::TreeBuilder which is more DOM-like.

Beyond that, you'll have to decide which elements are important and which are not based on what you're trying do to.

You may want to do some research yourself. Then, when you run into a problem, ask a question related to the problem. This sounds more like specification for a project that you want someone to do for you.

For starters, websites use tags for all sorts of things, and the problem is very complex. You would probably want to save information in h# and p tags, but you also may want to save div tag information if they use the id tag. In short, you'd have to write rules for each website you encounter, or employ some sort of fuzzy logic.

Instead of doing it on a tag by tag basis, why not try detecting sentences and grammar, or things likely to be in headings, and choose tags that include those things while stripping out the rest?

Here's my own tool to solve this problem in Perl using HTML::Parser as a github gist: html2txt.pl

It's unfinished and perhaps slightly Windows-centric but I thought I'd share it since a few people have viewed my question here. Feel free to play with it.