Batch UTF-8 Validation Tool?

Anyone know an app/service/method that I could use to validate a bunch of XML files for UTF-8?

Basically I have a ton of XML files that are suppose to be UTF-8 and some of them happen to contain some bogus characters causing them not to render right in the content viewer.

I know I can check one at a time with methods found in this answer: How to check whether a file is valid UTF-8?

...but how about thousands of XML files 开发者_如何学运维at once?

Why can't you take one of the solutions from the linked question and apply them to your situation? It seems like it'd be fairly simple to iterate over all the files you want to check, run iconv -f utf8 on them and emit a list of files where that fails.

Update
Since you haven't specified the situation or environment under which you need to do this test, it's hard to offer concrete advice. The post you linked offers methods of testing what you want, so it's just a matter of knowing what you have available to implement a solution.

Assuming a basic *nix envornment, this simple shell script provides a basic check, caveat the typical filename globbing issues.

#!/bin/sh
for f in *.xml; do
    if ! iconv -f utf8 $f >/dev/null 2>&1; then
        echo $f
    fi
done

Unless you provide more information about your specific requirements though, it's difficult to know whether any answers people have is actually relevant.

to expand on jamessan's answer using iconv, here is a modified shell script that you can use in conjunction with the unix find command to check all files matching some file extension regex pattern in a directory (including it's children)

#!/bin/sh

for i in "$@"
do

    if ! iconv -f utf8 $i >/dev/null 2>&1;
    then
            echo "failed: $i"
    #else
        #   echo "ok: $i"
    fi

done

say you name your script check_UTF8.sh, you can call it like so:

$ find -E . -type f -iregex ".*(.js|.css|.php|.tpl|.html)$" | xargs /path/to/check_UTF8.sh

all the files that match the file extension pattern regex (in this case, .js/.css/.php/.tpl/.html) are piped into the check_UTF8.sh script, and any file that have invalid UTF-8 is echo'd out in the form of it's full path, relative to the directory you are calling the script from

This simple python script also helps

def check(filename: str):
print(f"checking file: {filename} ")
with open(filename,"rb") as f:
    i = 0
    for line in f:
        try:
            test = line.decode("utf-8", errors="strict")
        except UnicodeDecodeError as e:
            print(f"{filename}:{i}: Error at {e.begin}~{e.end}, right after {line[0:e.begin].decode('utf-8')}")
        i += 1

import os

path = "C:/path/to/test/files/"

for subdir, dirs, files in os.walk(path):
    for filename in files: 
        check(os.path.join(subdir, filename))