How to change the internal char type of the parser?

I've been using spirit classic for quite a while and recently start learning v2. Both are easy to use and powerful enough to handle moderate parsing issue.

In those idyllic days with the spirit classic and ascii characters, the job is simply adapting an EBNF to LL form. But finally, I have to deal with the i18n.

My thought is really crude. Make a forward iterator which iterates over the input text byte stream of any encoding. The iterator handles these encoding conversion job and feeds the parser with utf16/32 code unit(the iterator could be easily implemented by either iconv or icu4c).

The type开发者_StackOverflow社区 of code unit should be the internal char type processed by parser. But after reading through the doc, I could find only several primitive, iterator templates have a explicit char_t parameter. Does that mean I have to reformulate those numerics, directives, scanners,and etc.?

I also checked out the v2 doc. It offers a namespace way to make everything consistent, but still not too much about how to explicitly change the internal char type. Again, I searched the mailing list archive, but seems those unicode and other encoding related post is still up in the air. Some one told me that spirit could still work through those files with different encoding. So I tested the parser using files with different encoding but same content. Several MBCS encoded files passed the test, and casually some utf8 file also passed. But other encoding failed most of the time.

I suspect you have already found char_ and character encoding namespace from the Boost Spirit web site.

The last page includes the helpful comment

"We have a namespace for each character set Spirit will be supporting. That includes ascii, iso8859_1, standard and standard_wide (and in the future, unicode)."

In other words boost spirit doesn't really support unicode at the moment. It's on their TODO list.

In the mean time you could try a kludgy workaround like this:

my_tag_ = lit("<") >> byte_ >> lit(">");

which will match any binary string which happens to appear between angle brackets, providing you don't use any charset dependant filter on it.