htmlcxx c++ crawling html

I've read many different issues from different people regarding libraries to crawl HTML. I've decided to go with htmlcxx, because it looks simple and it's in the Ubuntu repository.. Anyhow, while playing with htmlcxx I was trying to achieve a simple task and grabbing the text between heading tags. Using the iterator, it->text() returns the tag itself and it->textClosing() returns the closing of the tag. My question is, how can I get at the data BETWEEN the tags? I mean, there must be a way, why make a library to crawl html and not have this functionality? If anyone can point me in the right direction, I'd appreciate it.

You can check out what I've done with svn so far with: svn co svn://yunices.dyndns.org/repository/nich/trunk .

or view through websvn: https://yunices.dyndns.org/

Here's the particular snippet in question:

void node::get_headings() {
    tree<htmlcxx::HTML::Node>::iterator it = dom.begin();
    tree<htmlcxx::HTML::Node>::iterator end = dom.end();
    for (; it != end; ++it) {
        static const boost::regex expression("[h|H][1-6]");
        if(boost::regex_search(it->tagName(), expression)) {
            it->parseAttributes();
            std::cout << it->text() << "<=>" << it->closingText() << std::endl;
            std::map<std::string, std::string> pairs = it->attributes();
            for ( std::map<std::string, std::string>::const_iterator iter = pairs.begin();
               开发者_如何学Python     iter != pairs.end(); ++iter ) {
                std::cout << iter->first << ":" << iter->second << "\n";
            }
        }
    }
}

In most DOM libraries (and so in htmlcxx if I read the code correctly) the text of a tag is actually a node (or in the case of something like

<p> bla <p>blubb</p> blah </p>

more than one node).

You just have to iterate over all children of the tag and check that it's neither a comment nor a tag.

You can add this method to Node.h to get the contents contained between the tags (passing the original html string as an argument):

inline unsigned int contentLength() const { this->mLength - this->mText.length() - this->mClosingText.length(); }
inline std::string content(const std::string& html) const { return html.substr(this->mOffset + this->mText.length(), this->contentLength()); }

That works nicely Dave, thanks, actually there was a bracket missing, I just threw it into one line instead.

inline std::string content(const std::string& html) const { return html.substr(this->mOffset + this->mText.length(), this->mLength - (this->mText.length() + this->mClosingText.length())); }

The following function demonstrates a method of accessing child content.

std::string get_child_content( tree<HTML::Node> const & dom, tree<HTML::Node>::iterator const & parent )
   {
   std::string result;
   for ( unsigned i=0; i<dom.number_of_children(parent); i++ )
      {
      tree<HTML::Node>::iterator it = dom.child(parent,i);
      if ( !it->isTag() && !it->isComment() ) result += it->text();
      }
   return result;
   }

Keep in mind however, that as @filmor has pointed out, it is possible for HTML to represent multiple levels of descendants for any tag. The function that I have provided only captures the direct children.

Here's an example of how you might use this and the effect on some example HTML...

cout << it->text(); // display the opening tag
cout << get_child_content(dom,it); // display the contents
cout << it->closingText(); // display the closing tag

Raw HTML...

<h2>hello <span>w</span>orld</h2>

Resulting output (notice that span and its contents are missing)...

<h2>hello orld</h2>