Check valid string encoding with PHP's intl (ICU) features
Using the features currently available in PHP's intl wrapper for ICU, how would you go about checking for validity of a stri开发者_运维知识库ng's encoding? (e.g. check for valid UTF-8)
I know it can be done with mbstring, iconv() and PCRE but I'm specifically interested in intl with this question.
UConverter can be used Since PHP 5.5. The manual doesn't exist. See https://wiki.php.net/rfc/uconverter for API.
function replace_invalid_byte_sequence($str)
{
return UConverter::transcode($str, 'UTF-8', 'UTF-8');
}
function replace_invalid_byte_sequence2($str)
{
return (new UConverter('UTF-8', 'UTF-8'))->convert($str);
}
function utf8_check_encoding($str)
{
return $str === UConverter::transcode($str, 'UTF-8', 'UTF-8');
}
function utf8_check_encoding2($str)
{
return $str === (new UConverter('UTF-8', 'UTF-8'))->convert($str);
}
// Table 3-8. Use of U+FFFD in UTF-8 Conversion
// http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf)
$str = "\x61"."\xF1\x80\x80"."\xE1\x80"."\xC2"."\x62"."\x80"."\x63"
."\x80"."\xBF"."\x64";
$expected = 'a���b�c��d';
var_dump([
$expected === replace_invalid_byte_sequence($str),
$expected === replace_invalid_byte_sequence2($str)
],[
false === utf8_check_encoding($str),
false === utf8_check_encoding2($str)
]);
I did some digging and found ICU unorm2_normalize() documentation. Its pErrorCode out parameter is interesting. The standard ICU error codes start around line 620 of utypes.h. So I tried this test script:
$s = 'tête-à-tête';
echo "normalizer_normalize(\$s) >> "
. var_export(normalizer_normalize($s), 1) . "\n";
$s = "\xFF" . $s;
echo "normalizer_normalize(\$s) >> "
. var_export($r=normalizer_normalize($s), 1) . "\n";
if ($r===false)
echo "normalizer_normalize() error: "
. intl_get_error_message() . "\n";
// which outputs:
normalizer_normalize($s) >> 'tête-à-tête'
normalizer_normalize($s) >> false
normalizer_normalize() error: Error converting input string to UTF-16: U_INVALID_CHAR_FOUND
So I guess a test based on that and looking for the following three error codes would be a decent indication of bad UTF-8 encoding:
U_INVALID_CHAR_FOUND Character conversion: Unmappable input sequence. U_TRUNCATED_CHAR_FOUND Character conversion: Incomplete input sequence. U_ILLEGAL_CHAR_FOUND Character conversion: Illegal input sequence/combination of input units.
Or when I'm feeling lazy I could just use
normalizer_normalize($s)===false
Btw: I'm confused by this line of the ICU API spec:
pErrorCode Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
The "the function returns immediately" phrase is encouraging re performance of my test but does "the function" refer to unorm2_normalize() or U_SUCCESS()? Any ideas?
精彩评论