Using python 2.7, why does my unicode filename raise an IOError when calling file() on it?

Python 2.7:

I am trying to open an mp3 to read its ID3 tag using mutagen (so I'd rather not have to change the method), which calls:

file(filename, "rb")

For files with no special characters this works fine, but otherwise, I sometimes seem to get

IOError: [Errno 2] No such file or directory: u"somepath\\08 - Muse - I Belong To You - Mon C\x9cur S'ouvre \xc0 Ta Voix.mp3"

while other times

u"somepath\\02 - Max\xefmo Park - Apply Some Pressure.mp3"

works fine.

What's the difference between the two? Why does one work while the other doesn't?

Cheers,

Felix

EDIT: It worked when running under pydev in Eclipse, for which

sys.getdefaultencoding()

returned "Cp1252" but not from the command line which returned "ascii". The filename when printed to the Eclipse console was

u"somepath\\08 - Muse - I Belong To You - Mon C\u0153ur S'ouvre \xc0 Ta Voix.mp3"

EDIT: The code which gets the filename from Winamp (the music player) is:

winampProcess = win32api.OpenProcess(win32con.PROCESS_VM_READ, False, processID)
memoryBuffer = ctypes.create_string_buffer(256)
ctypes.windll.kernel32.ReadProcessMemory(winampProcess.handle, memoryPointer, memoryBuffer, 256, 0)
winampProcess.Close()
rawPath = win32api.GetFullPathName(memoryBuffer.raw.split("\x00")[0])
try:
    unicodeString = unicode(rawPath)
except UnicodeDecodeError:
    unicodeString = u""
    for char in rawPath:
        try:
            unicodeString += unicode(char)
        except UnicodeDecodeError as err:
            errStr = str(err)
            startIndex = errStr.index("0x")
            endIndex = errStr.index(" ", startIndex)
            hexStr = ""
            for i in range(startIndex, endIndex):
                hexStr += errStr[i]
            unicodeString += unichr(int(hexStr, 16))
return unicodeString

EDIT: The problem is fixed if I explicitly set

unicode(str, "cp1252")

but I still don't 开发者_StackOverflow中文版understand what causes the problem, and this is a hacky fix that probably won't work for other dodgy filenames...

Just guessing - you're pulling the filename from a program which is using a multibyte character set in the current default encoding, which is cp1252 for English versions of Windows. Ascii doesn't include any extended characters, which is why you get the error when you try to encode the string into Unicode using the Ascii encoding.

Edit: this answer has some information about encoding file names in the current Windows code page.

Use os.listdir() on the directory to see what the filename is, encoded. Then compare that to what you get when you do filename.encode('cp1252'). There should be a difference, and that should tell you what is wrong.

The only real problem I can think of is that something gets decoded twice. You could have normalization problems too, but that seems unlikely in this case.