Can python.org releases play nice with Apple framework builds?

(Advanced apologies for the lack or proper links; the system won't allow me to add more than two.)

Unfortunately, I've learnt the hard way that you shouldn't mess with the default Python installations in Mac OS X (specifically, 10.6.8).

After using the python.org installers for 2.6.6 (http://www.python.org/getit/releases/2.6.6/) and 2.5.4 (http://www.python.org/getit/releases/2.5.4/), I have Python versions which are more mature than those provided by Apple (which is great for development), but have broken core system functio开发者_开发技巧nality (which is bad for just about everything else.) The most visible breaks so far have been when trying to run namebench (https://code.google.com/p/namebench/), Blink (http://icanblink.com/) and Mercurial (https://www.mercurial-scm.org/). From what I can gather, it's down to the paths.

In a default installation, the paths should resemble something those outlined in this question. Instead, mine look like this:

$ /usr/bin/python2.6 -V
Python 2.6.6

$ which python
/usr/bin/python
$ python
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for i in sys.path:
...     print i
... 
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pip-1.0.2-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

(It's a similar story for 2.5 but, for simplicity, I'll stick with 2.6.)

The issue seems to be the non-inclusion of:

• /Library/Python/2.6/site-packages/*.egg
• /Library/Python/2.6/site-packages
• /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
• /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python

This explains why Blink and namebench can't find PyObjC (Is PyObjC pre-installed on OSX SL?) and Mercurial can't find its modules (in /Library/Python/2.6)

Issue #4865 in the Python bug tracker partially addresses this particular problem, which has since been fixed for versions 2.7 and greater. But because it's "a feature request and not a bugfix", has not been back-ported to 2.5 and 2.6. Even then, looking at the committed fix (http://hg.python.org/lookup/r70778), I'm not sure it addresses the lack of references to the "Extras" directories.

I understand that I can manually add paths to Python by manually altering site.py. I could also make use of the PYTHONPATH environment variable. I'd rather not cause any further damage by altering site.py, and changes PYTHONPATH would only be valid for scripts/applications run from the shell using my user account.

Can I get these more recent versions of Python to reference the same paths as the default framework installations? If so, what is the best way to go about it? If not, is there an accepted method of rolling back to the system defaults?

You misunderstand how Python installations work on OS X. Each Python instance has its own site-packages directory. The standard location for framework installers is within the framework at ./lib/pythonx.y/site-packages. So for the python.org installers which install into /Library/Frameworks/Python.framework, you will find its 2.6 site-packages here:

/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/

Apple makes some modifications to the versions of Python that it ships with OS X releases. From OS X 10.5 on, the system Pythons are installed at:

/System/Library/Frameworks/Python.framework

and Apple chooses to include some extra 3rd-party packages in the non-standard ./Extras directory within each version. It also uses a non-standard location for each version's site-packages directory. They are created in /Library/Python/, presumably so that user-installed packages do not modify anything in /System/Library. So for the Apple-supplied Python 2.6, its site-packages directory is:

/Library/Python/2.6/site-packages

and can be thought of as having been extended by the packages in ./Extras.

Each Python instance has a separate site-packages directory. It is not intended and often not possible to share packages across site-packages of different instances even of the same minor version of Python, i.e. 2.6. The most obvious problem is that there are often differences in the C compiler version, the OS X ABI (MACOSX_DEPLOYMENT_TARGET), the SDK version, and/or the CPU architectures used to build the Python interpreters and which are subsequently used by Distutils to build C extension modules included in 3rd-party packages.

In Mac OS X 10.6, the Apple-supplied Python is built using gcc-4.2 and targeted just for OSX 10.6 and includes 3 CPU architectures (i386, x86_64, and ppc). The python.org installers for Python 2.6 are built to run on older systems as well, so have a target of 10.3 and later, use gcc-4.0, and are 32-bit only (i386 and ppc). So, in general, you cannot run C extension modules built for the one Python with the other.

This means that, in general, you need to install separate copies of 3rd-party packages you need with and for each Python, if they are not already included with that Python. This includes basic items like easy_install (from setuptools or Distribute). The system Pythons in 10.5+ include easy_install version in /usr/bin for them. If you install a python.org Python, you'll need to install a separate version for it; by default, the easy_install command will be installed in that Python's ./bin directory in its framework; that is the Distutils default location. That's why it is recommended that you add this directory to your shell PATH (and the python.org installer for Python 2 do that automatically by default).

The change introduced by Issue4865 is not really a good solution and may fail with C extension modules. I would not depend on it remaining in Python in future versions.

Also installing a python.org Python in no way breaks anything in the system Python because they are completely independent installations using different file system locations. The only thing that may change is which instance of Python is invoked when you type a particular name. That is controlled primarily by the search order of your shell PATH environment variable. As noted, the python.org installer by default changes that order but the system Python is still readily available by using its absolute path /usr/bin/python2.6. Or you can revert the changes to shell profile, for instance, .bash_profile.

$ echo $PATH
/Library/Frameworks/Python.framework/Versions/2.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/Library/Frameworks/Python.framework/Versions/2.6/bin/python
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
$ python -V
Python 2.6.6
$ python2.6 -V
Python 2.6.6
$ /usr/bin/python -V
Python 2.6.1
$ /usr/bin/python2.6 -V
Python 2.6.1
#
# remove python.org Python 2.6 from PATH
#
$ export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/usr/bin/python
/usr/bin/python2.6
$ python -V
Python 2.6.1
$ python2.6 -V
Python 2.6.1