Questions while updating some scanner code to use ICU
I am working on a rudimentary hand-coded lexical scanner and wish to support UTF-8 input (it's not 1970 anymore!). Input characters are read from stdin
or a file one at a time and pushed into a buffer until whitespace is seen, etc. I thought about writing my own wrapper for fgetc()
that would instead return char[]
of bytes that make up the UTF-8 character and work with the result as a string... it'd be easy enough, but would become a slippery-slope. I'd rather not waste time re-inventing the wheel and instead use an existing, tested library like ICU. And so now I have a non-UTF-8 supporting code that works with fgetc()
, isspace()
, strcmp()
, etc. which I am trying to update to use ICU. This is my first foray with ICU and have been reading through the documentation and trying to find usage examples with Google code search, but there are still some points of confusion I'm hoping开发者_运维百科 someone will be able to clarify.
The u_fgetc()
function returns UChar
, and u_fgetcx()
returns UChar32
... the documentation recommends using u_fgetcx()
to read codepoints, so that's what I'm starting with. I'm keeping the same approach as above, but I'm pushing UChar32
s into a buffer instead of char
s.
What is the proper way to compare a character against a known value? Originally I was able to do
if (c == '+')
to check if the plus-sign was fetched from the input. GCC doesn't complain whenc
is aUChar32
(which is then a comparison betweenUChar32
andchar
) but is this really proper?I was able to use
strcmp()
to compare the buffered characters with a known value, for exampleif ((strcmp(buf, "else") == 0)
. There isu_strcmp()
provided by ICU and I think I may need to use theU_STRING_DECL
andU_STRING_INIT
macros to specify the known literal, but I am not certain. The documentation shows they result inUChar[]
, though I assume I needUChar32[]
... and I'm uncertain how to use them correctly anyway. Any guidance here would be welcomed.After reading in a series of numeric characters I have been converting them with
strtol()
so I can work with them. Is there a similar function made available by ICU since I am convertingUChar32[]
now?
UChar
is for holding a Code Unit, while UChar32
is for holding a Code Point. If your input stays on the Basic Multilingual Plane (BMP), UChar
is sufficient, and indeed most ICU functions operate on UChar[]
.
Strongly recommended reading is the ICU User Guide, which explains most of the internals and best practices.
What is the proper way to compare a Unicode character variable against a known value? A character (or
UChar
orUChar32
) is just another integer type with a certain width and signedness, and can be compared to other integer types with the usual caveats and restrictions. As for defining a character value, C99 (chapter 6.4.3) provides Universal character names notation:\u
followed by four hex digits, or\U
followed by eight hex digits, specifying the ISO/IEC 10646 "short identifier". The area below 0x00a0 (with exceptions of 0x0024'$'
, 0x0040'@'
, and 0x0060 (backtick) is reserved (but can be represented by casting a simple character constant toUChar
). Also reserved is the range from 0xd800 through 0xdfff (for use by UTF-16).How to define Unicode string literals?
U_STRING_DECL
andU_STRING_INIT
are indeed what you're looking for. (As written above, ICU mainly operates onUChar[]
.) If you were using C++ instead of C,UNICODE_STRING_SIMPLE
(optionally followed bygetTerminatedBuffer()
to yieldUChar[]
again) provides a much more comfortable way of defining Unicode string literals.How to convert a Unicode string representing a numerical into that numerical's value?
unum_parse()
and its brethren inunum.h
will help you there.
The Unicode value for PLUS SIGN is U+002B, and the normal (Latin-1) value for '+' is also 0x2B (053, 43). What you wrote is safe enough where the code set is based on ASCII or ISO-8859-x. The C99 standard provides for Unicode (Universal character names) of the forms
\u0123
and\U00102345
(with 4 and 8 hexadecimal digits), but stipulates that you cannot specify values less than\u00A0
, such as\u002B
. So, I think what you wrote is correct.However, you could save yourself future angst by using an
enum
such asenum { PLUS_SIGN = '+' };
defined in an appropriate header and used whereever you need a literal plus sign. That way, if your assumption (and my assumption) is wrong, you have one place to edit - the header.
I note that the page on Strings with ICU suggests that using UTF-32 in an application is unusual.
In pure C, you'd probably use
wcscmp(buf, L"else")
, assuming that thewchar_t
on your system is equivalent touint32_t
and/or UChar32. There seem to be ways to useUnicodeString
andUNICODE_STRING("...")
followed byToUTF32()
to create a UTF-32 string. There may also be neater ways.There are 'Formatting' classes which handle both formatting and parsing. You would probably use classes derived from the
NumberFormat
class.
精彩评论