Unicode filenames on Windows with Python & subprocess.Popen()

Why does the following occur:

>>> u'\u0308'.encode('mbcs')   #UMLAUT
'\xa8'
>>> u'\u041A'.encode('mbcs')   #CYRILLIC CAPITAL LETTER KA
'?'
>>>

I have a Python application accepting filenames from the operating system. It works for some international users, but not others.

For example, this unicode filename: u'\u041a\u0433\u044b\u044b\u0448\u0444\u开发者_开发百科0442'

will not encode with Windows 'mbcs' encoding (the one used by the filesystem, returned by sys.getfilesystemencoding()). I get '???????', indicating the encoder fails on those characters. But this makes no sense, since the filename came from the user to begin with.

Update: Here's the background to my reasons behind this... I have a file on my system with the name in Cyrillic. I want to call subprocess.Popen() with that file as an argument. Popen won't handle unicode. Normally I can get away with encoding the argument with the codec given by sys.getfilesystemencoding(). In this case it won't work

In Py3K - at least from Python 3.2 - subprocess.Popen and sys.argv work consistently with (default unicode) strings on Windows. CreateProcessW and GetCommandLineW are used obviously.

In Python - up to v2.7.2 at least - subprocess.Popen is buggy with Unicode arguments. It sticks to CreateProcessA (while os.* are consistent with Unicode). And shlex.split creates additional nonsense.

Pywin32's win32process.CreateProcess also doesn't auto-switch to the W version, nor is there a win32process.CreateProcessW. Same with GetCommandLine. Thus ctypes.windll.kernel32.CreateProcessW... needs to be used. The subprocess module perhaps should be fixed regarding this issue.

UTF8 on argv[1:] with private apps remains clumsy on a Unicode OS. Such tricks may be legal for 8-bit "Latin1" string OSes like Linux.

UPDATE vaab has created a patched version of Popen for Python 2.7 which fixes the issue.
See https://gist.github.com/vaab/2ad7051fc193167f15f85ef573e54eb9
Blog post with explanations: http://vaab.blog.kal.fr/2017/03/16/fixing-windows-python-2-7-unicode-issue-with-subprocesss-popen/

DISCLAIMER: I'm the author of the fix mentionned in the following.

To support unicode command line on windows with python 2.7, you can use this patch to subprocess.Popen(..)

The situation

Python 2 support of unicode command line on windows is very poor.

Are severly bugged:

• issuing the unicode command line to the system from the caller side (via subprocess.Popen(..)),

• and reading the current command line unicode arguments from the callee side (via sys.argv),

It is acknowledged and won't be fixed on Python 2. These are fixed in Python 3.

Technical Reasons

In Python 2, windows implementation of subprocess.Popen(..) and sys.argv use the non unicode ready windows systems call CreateProcess(..) (see python code, and MSDN doc of CreateProcess) and does not use GetCommandLineW(..) for sys.argv.

In Python 3, windows implementation of subprocess.Popen(..) make use of the correct windows systems calls CreateProcessW(..) starting from 3.0 (see code in 3.0) and sys.argv uses GetCommandLineW(..) starting from 3.3 (see code in 3.3).

How is it fixed

The given patch will leverage ctypes module to call C windows system CreateProcessW(..) directly. It proposes a new fixed Popen object by overriding private method Popen._execute_child(..) and private function _subprocess.CreateProcess(..) to setup and use CreateProcessW(..) from windows system lib in a way that mimics as much as possible how it is done in Python 3.6.

How to use it

How to use the given patch is demonstrated with this blog post explanation. It additionally shows how to read the current processes sys.argv with another fix.

Docs for sys.getfilesystemencoding() say that for Windows NT and later, file names are natively Unicode. If you have a valid unicode file name, why would you bother encoding it using mbcs?

Docs for codecs module say that mbcs encodes using "ANSI code page" (which will differ depending on user's locale) so if the locale doesn't use Cyrillic characters, splat.

Edit: So your process is calling subprocess.Popen(). If your invoked process is under your control, the two processes ahould be able to agree to use UTF-8 as the Unicode Transport Format. Otherwise, you may need to ask on the pywin32 mailing list. In any case, edit your question to state the degree of control you have over the invoked process.

If you need to pass the name of an existing file, then you might have a better chance of success by passing the 8.3 version of the Unicode filename.

You need to have the pywin32 package installed, then you can do:

>>> import win32api
>>> win32api.GetShortPathName(u"C:\\Program Files")
'C:\\PROGRA~1'

I believe these short filenames use only ASCII characters, and therefore you should be able to use them as arguments to a command line.

Should you need to specify also filenames to be created, you can create them with zero size in advance from Python using Unicode filenames, and pass the short name of the file as an argument.

UPDATE: user bogdan says correctly that 8.3 filename generation can be disabled (I had it disabled, too, when I had Windows XP on my laptop), so you can't rely on them. So, as another more far-fetched approach when working on NTFS volumes, one can hard link the Unicode filenames to plain ASCII ones; pass the ASCII filenames to an external command and delete them afterwards.

With Python 3, just don't encode the string. Windows filenames are natively Unicode, and all strings in Python 3 are Unicode, and Popen uses the Unicode version of the CreateProcess Windows API function.

With Python 2.7, the easiest solution is to use the third-party module https://pypi.org/project/subprocessww/. There is no "built-in" solution to get full Unicode support (independent of system locale), and the maintainers of Python 2.7 consider this a feature request rather than a bugfix, so this is not going to change.

For a detailed technical explanation of why things are as they are, please see the other answers.