Japanese mojibake detection

I want to know if there is a way to detect mojibake (Invalid) ch开发者_开发技巧aracters by their byte range. (For a simple example, detecting valid ascii characters is just to see if their byte values are less 128) Given the old customized characters sets, such as JIS, EUC and of course, UNICODE, is there a way to do this?

The immediate interest is in a C# project, but I'd like to find a language/platform independent solution as much as possible, so I could use in c++, Java, PHP or whatever.

Arrigato

detecting 文字化け(mojibake) by byte range is very difficult.

As you know, most Japanese characters consist of multi-bytes. In Shift-JIS (one of most popular encodings in Japan) case, the first-byte range of a Japanese character is 0x81 to 0x9f and 0xe0 to 0xef, and the second-byte has other range. In addition, ASCII characters may be inserted into Shift-JIS text. it's difficult.

In Java, you can detect invalid characters with java.nio.charset.CharsetDecoder.

What you're trying to do here is character encoding auto-detection, as performed by Web browsers. So you could use an existing character encoding detection library, like the universalchardet library in Mozilla; it should be straightforward to port it to the platform of your choice.

For example, using Mark Pilgrim's Python 3 port of the universalchardet library:

>>> chardet.detect(bytes.fromhex('83828357836f8350'))
{'confidence': 0.99, 'encoding': 'SHIFT_JIS'}
>>> chardet.detect(bytes.fromhex('e383a2e382b8e38390e382b1'))
{'confidence': 0.938125, 'encoding': 'utf-8'}

But it's not 100% reliable!

>>> chardet.detect(bytes.fromhex('916d6f6a6962616b6592'))
{'confidence': 0.6031748712523237, 'encoding': 'ISO-8859-2'}

(Exercise for the reader: what encoding was this really?)

This is not a direct answer to the question, but I've had luck using the ftfy Python package to automatically detect/fix mojibake:

• https://github.com/LuminosoInsight/python-ftfy
• https://pypi.org/project/ftfy/
• https://ftfy.readthedocs.io/en/latest/

>>> import ftfy
>>> print(ftfy.fix_encoding("(ง'⌣')ง"))
(ง'⌣')ง

It works surprisingly well for my purposes.

I don't have time and / or priority level to follow up on this for the moment, but I think, if knowing the source is Unicode, using these charts and following on some of the work done here, I think some headway can be made into the issue. Likewise, for Shift-JIS, using this chart can be helpful.