开发者

Check valid string encoding with PHP's intl (ICU) features

Using the features currently available in PHP's intl wrapper for ICU, how would you go about checking for validity of a stri开发者_运维知识库ng's encoding? (e.g. check for valid UTF-8)

I know it can be done with mbstring, iconv() and PCRE but I'm specifically interested in intl with this question.


UConverter can be used Since PHP 5.5. The manual doesn't exist. See https://wiki.php.net/rfc/uconverter for API.

function replace_invalid_byte_sequence($str)
{
    return UConverter::transcode($str, 'UTF-8', 'UTF-8');
}

function replace_invalid_byte_sequence2($str)
{
    return (new UConverter('UTF-8', 'UTF-8'))->convert($str);
}

function utf8_check_encoding($str)
{
    return $str === UConverter::transcode($str, 'UTF-8', 'UTF-8');
}

function utf8_check_encoding2($str)
{
    return $str === (new UConverter('UTF-8', 'UTF-8'))->convert($str);
}

// Table 3-8. Use of U+FFFD in UTF-8 Conversion
// http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf)
$str =  "\x61"."\xF1\x80\x80"."\xE1\x80"."\xC2"."\x62"."\x80"."\x63"
    ."\x80"."\xBF"."\x64";
$expected = 'a���b�c��d';

var_dump([
    $expected === replace_invalid_byte_sequence($str),
    $expected === replace_invalid_byte_sequence2($str)
],[
    false === utf8_check_encoding($str),
    false === utf8_check_encoding2($str)
]);


I did some digging and found ICU unorm2_normalize() documentation. Its pErrorCode out parameter is interesting. The standard ICU error codes start around line 620 of utypes.h. So I tried this test script:

$s = 'tête-à-tête';
echo "normalizer_normalize(\$s) >> " 
     . var_export(normalizer_normalize($s), 1) . "\n";
$s = "\xFF" . $s;
echo "normalizer_normalize(\$s) >> " 
     . var_export($r=normalizer_normalize($s), 1) . "\n";
if ($r===false)
    echo "normalizer_normalize() error: " 
         . intl_get_error_message() . "\n";
// which outputs:
normalizer_normalize($s) >> 'tête-à-tête'
normalizer_normalize($s) >> false
normalizer_normalize() error: Error converting input string to UTF-16: U_INVALID_CHAR_FOUND

So I guess a test based on that and looking for the following three error codes would be a decent indication of bad UTF-8 encoding:

U_INVALID_CHAR_FOUND Character conversion: Unmappable input sequence. U_TRUNCATED_CHAR_FOUND Character conversion: Incomplete input sequence. U_ILLEGAL_CHAR_FOUND Character conversion: Illegal input sequence/combination of input units.

Or when I'm feeling lazy I could just use

normalizer_normalize($s)===false

Btw: I'm confused by this line of the ICU API spec:

pErrorCode Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)

The "the function returns immediately" phrase is encouraging re performance of my test but does "the function" refer to unorm2_normalize() or U_SUCCESS()? Any ideas?

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜