Python unicode works in 2.6.1 on OSX, but not in 2.6.5 on Ubuntu

Given the following code run from the Python interpreter:

import sys
sys.getdefaultencoding()
my_string = '\xc3\xa9'
my_string = unicode(my_string, 'utf-8')
my_string
print my_string

With Python 2.6.1 running on a mac, everything works fine:

$ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
&开发者_JS百科gt;>> sys.getdefaultencoding()
'ascii'
>>> my_string = '\xc3\xa9'
>>> my_string = unicode(my_string, 'utf-8')
>>> my_string
u'\xe9'
>>> print my_string
é
>>> 

With Python 2.6.5 running on Ubuntu 10.04 LTS, it fails:

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> my_string = '\xc3\xa9'
>>> my_string = unicode(my_string, 'utf-8')
>>> my_string
u'\xe9'
>>> print my_string
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)
>>> 

Has something changed between Python 2.6.1 and 2.6.5 that requires different handling of unicode strings? Or does this have to do with something misconfigured in my (default Ubuntu server 10.04 LTS) linux environment?

Edit: Both environments have LANG=en_US.UTF-8

It can happen with C locale. Try running Python with LANG=en_US.UTF-8 python and trying your code again.

I can reproduce the error with the command:

$ PYTHONIOENCODING=ascii python -c'print "\xc3\xa9".decode("utf-8")'

Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0:\
ordinal not in range(128)

sys.getdefaultencoding() is 'ascii' and not very useful by default.

Try using your console encoding:

$ PYTHONIOENCODING=utf-8 python -c'print "\xc3\xa9".decode("utf-8")'
é

or

$ python -c'import locale; print "\xc3\xa9".decode("utf-8").encode(
> locale.getpreferredencoding())'
é

Check sys.stdout.encoding:

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding'
True UTF-8

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding' | cat
False None

$ python -c'import sys; o = sys.stdout; print o.isatty(), o.encoding' >/tmp/out
$ cat /tmp/out
False None

If sys.stdout.encoding is None try to use locale.getpreferredencoding() or set PYTHONIOENCODING as shown above. See http://wiki.python.org/moin/PrintFails

If the error occurs only in the interactive Python session then look at sys.displayhook().

Have you tried prefixing your string with u?

my_string = u'\xc3\xa9'

See http://docs.python.org/howto/unicode.html#unicode-literals-in-python-source-code

In Python source code, Unicode literals are written as strings prefixed with the ‘u’ or ‘U’ character: u'abcdefghijk'. Specific code points can be written using the \u escape sequence, which is followed by four hex digits giving the code point. The \U escape sequence is similar, but expects 8 hex digits, not 4.

• Python 3.6.8 or newer

As @jfs answered,

$ PYTHONIOENCODING=utf-8 python file.py 

worked for me. And if you want to make it default you can add following command to your basrc or zshrc

export PYTHONIOENCODING="utf-8"