Decryption of text file with format other than ASCII

I've been given an encrypted file where the plaintext has a "common (but not extremely common these days)" format. ~80000 bytes

It has been encrypted with what I would describe as a Vigenere cipher with modified encryption table. One byte of the key and one byte of the plaintext map to one byte of the ciphertext. The keystring has a certain length, so the key character used for encryption cycles through the keystring.

The key contains only alphanumeric characters.

So far I've determined that the keylength is 30/60 by finding the least common divisor of the start position of repeated triples in the ciphertext. Pretty standard for Vigenere.

Now, I've been guessing at possible characters for the key by observing what the decrypted bytes would be and eliminating possibilities if they fall outside of acceptable ranges (since 32-126 are visible, no values between 16-31, etc.)

This worked for the first part, which had a small key and the plaintext was straight ASCII.

When I try it for the larger file and "new file format" it rejects all possible characters.

This eliminates ASCII, Ascii85, Base64, windows-1252, utf-7, QP and uuencode as they all rely on the ASCII character set. I've also made filters for EBCDIC and ISO8859-1 which rejected all keys. Utf-8 also failed because no key would make all bytes begin with either 0,10,110,1110,11110,111110 or 1111110.

The remaining character encodings I haven't tried that I suspect are UTF-16,32,1 which I'm unsure of how to filter.

My questions are:

• are there other character encodings that I am forgetting?
• Is it possible that I'm filtering out too much and that some out of bound characters should be allowed to slide through?
• Could file format mean something other than character encoding? if so, how do I reconcile that with the filter still breaking on ASCII characters?
• What if the file format means compressed or archived?

Here is the filtering code I use, the filter is variable depending on what I'm trying to sift out.

void guessCrypt(string fileName, int keyLength, int index)
{
    byte[] file = cast(byte[])read(fileName);
    foreach(key;ValidKeyChars)
    {
        bool work = true;
        for(int x = index; x < file.length-10; x+=keyLength)
        {
            byte single = file[x]开发者_开发百科;
            int res = sdecrypt(single,key);
            if ((res < 32 && res > 15) || res > 126) //FILTER - this one ASCII
            {
                work = false;
                break;
            }
        }
        if (work == true)
        {
            writefln("\nwork: %s",key);
        }
    }

}

I would suggest not even bothering with existing encodings; treat it as another layer of a substitution cipher, and work out based on letter frequencies what maps to what. If the characters are, in fact, contiguous, that will only help your analysis. And once you know what they really map to, you can work from there to find what character set it, in fact, was.