Getting Python under control on Mac OS X - setting up environment and libraries

After starting o开发者_JAVA百科ut with Python on Ubuntu Linux, I've now for a good while been doing most of my sustained work on the Mac, currently Mac OS X 10.6. Unfortunately I've neglected to give proper attention to how Python is installed there and ended up with:

• Python 2.6.1 (Mac default version?) in /usr/bin (also, 2.5.4, which I'm not sure how it got there)
• Python 2.6.5 installed via MacPorts in /opt/local/bin/. This is my default
• I use pip to install libraries, which end up in some ungodly place (something like /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ - SRSLY?)
• Otherwise, mostly TextMate, and Git for version control. Django and Google App Engine, etc...

Now I'm preparing to set up 2.7 and 3.2, and I am unhappy with the haphazard state of things. So what are your favourite approaches to organising the code and libraries, and how have you wrestled the Mac into submission?

I'd like to continue using pip, but would like to have more control or at least understanding of what libraries for which version are getting installed and made accessible from where: I've had problems with installing py.test via pip and only being able to load it from the obsolete 2.6.1 Python, not my current 2.6.5 one. MacPorts has python_select, but it's not overly helpful:

reason: chris$ python_select -l
Available versions:
current none python26 python26-apple

Most Python people I've asked don't use MacPorts, which I don't much like, but the stock Mac Python from python.org. I've also heard the recommendation to use virtualenv systematically, so what is the link to a good practical introduction?

virutalenv is a great tool and is very useful for managing multiple Python instances on most platforms. On Mac OS X, though, things are more complicated because the use of framework builds makes it common to encounter multiple instances of even the same major version of Python. I suggest you first understand and be comfortable with how to manage framework installs on Mac OS X before delving into virtualenv.

"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ - SRSLY?"

Yes, that's the default location within a Python Mac OS X framework build. Most distributors of Python on Mac OS X use a framework build, each generally using a different root, because it makes it simpler to support multiple versions and to keep all the site packages installed for a particular Python instance together in one place.

For Mac OS X 10.6, Apple supplies a Python 2.6 (rooted at /System/Library/Frameworks/Python.framework and symlinked to /usr/bin/python2.6) as well as a Python 2.5 (for compatibility with Mac OS X 10.5 - that's the /usr/bin/python2.5 you see). The python.org Mac OS X installers install to /Library/Frameworks/Python.framework. MacPorts uses /opt/local/Library/Frameworks/Python.framework.

The frameworks include a bin directory (at ../Versions/x.x/bin in the framework); this is normally the default location where package scripts will be installed. It's also where symlinks to the interpreter binaries are placed.

Here's a look at a python.org-style framework with multiple versions of Python installed:

$ cd /Library/Frameworks/Python.framework/Versions/
$ ls -l
total 8
drwxrwxr-x  8 root  admin  340 Dec 23  2008 2.5/
drwxrwxr-x  8 root  admin  340 Aug 10 21:42 2.6/
drwxrwxr-x  8 root  admin  340 Jul  3 12:24 2.7/
drwxrwxr-x  7 root  admin  306 Mar 23 17:42 3.1/
drwxrwxr-x  7 root  admin  306 Aug 10 22:14 3.2/
lrwxr-xr-x  1 root  admin    3 Aug 10 21:47 Current@ -> 2.6
$ cd 2.6
$ ls -l
total 7776
lrwxr-xr-x  1 root  admin       17 Aug 10 21:47 Headers@ -> include/python2.6
drwxrwxr-x  3 root  admin      102 Aug 10 21:42 Mac/
-r-xrwxr-x  1 root  admin  3973552 Aug 10 21:41 Python*
drwxrwxr-x  4 root  admin      170 Aug 10 21:48 Resources/
drwxrwxr-x  2 root  admin      578 Aug 12 15:19 bin/
drwxrwxr-x  3 root  admin      102 Aug 10 21:42 include/
drwxrwxr-x  3 root  admin      646 Aug 10 21:43 lib/
drwxrwxr-x  3 root  admin      102 Aug 10 21:41 share/
$ ls -l bin
total 264
-rwxrwxr-x  1 root  admin    150 Aug 10 21:41 2to3*
-rwxr-xr-x  1 nad   admin    360 Aug 12 15:19 easy_install*
-rwxr-xr-x  1 nad   admin    368 Aug 12 15:19 easy_install-2.6*
lrwxr-xr-x  1 root  admin      7 Aug 10 21:47 idle@ -> idle2.6
-rwxrwxr-x  1 root  admin    138 Aug 10 21:41 idle2.6*
lrwxr-xr-x  1 root  admin      8 Aug 10 21:47 pydoc@ -> pydoc2.6
-rwxrwxr-x  1 root  admin    123 Aug 10 21:41 pydoc2.6*
lrwxr-xr-x  1 root  admin      9 Aug 10 21:47 python@ -> python2.6
lrwxr-xr-x  1 root  admin     16 Aug 10 21:47 python-config@ -> python2.6-config
-rwxrwxr-x  1 root  admin  29584 Aug 10 21:43 python2.6*
-rwxrwxr-x  1 root  admin   1463 Aug 10 21:42 python2.6-config*
lrwxr-xr-x  1 root  admin     10 Aug 10 21:47 pythonw@ -> pythonw2.6
-rwxrwxr-x  1 root  admin  29584 Aug 10 21:43 pythonw2.6*
lrwxr-xr-x  1 root  admin     11 Aug 10 21:47 smtpd.py@ -> smtpd2.6.py
-rwxrwxr-x  1 root  admin  18272 Aug 10 21:41 smtpd2.6.py*

Here's a MacPorts bin including a number of additional site packages:

$ cd /opt/local/Library/Frameworks/Python.framework/Versions/2.6
$ ls -l bin
total 952
-rwxr-xr-x  2 root  wheel    160 Jul 15 01:56 2to3*
-rwxr-xr-x  2 root  wheel   1541 Jun  8 19:59 apirst2html.py*
-rwxr-xr-x  2 root  wheel    363 Oct  5  2009 bdist_mpkg*
[...]
-rwxr-xr-x  2 root  wheel    211 Jun  2 12:45 django-admin.py*
-rwxr-xr-x  2 root  wheel    410 Jul 15 17:30 easy_install*
-rwxr-xr-x  2 root  wheel    418 Jul 15 17:30 easy_install-2.6*
[...]
-rwxr-xr-x  2 root  wheel    368 Jun 24 02:00 pip*
-rwxr-xr-x  1 root  wheel    342 Sep 18  2009 pkginfo*
[...]
lrwxr-xr-x  1 root  wheel      9 Jul 15 01:58 python@ -> python2.6
lrwxr-xr-x  1 root  wheel     16 Jul 15 01:58 python-config@ -> python2.6-config
-rwxr-xr-x  2 root  wheel  24932 Jul 15 01:57 python2.6*
-rwxr-xr-x  2 root  wheel   1473 Jul 15 01:57 python2.6-config*
[...]

If you install 2.7 or 3.1 Pythons using python.org installers or MacPorts, they are all rooted under /Library/Frameworks or /opt/local/Library/Frameworks and will happily co-exist with other versions. To successfully use framework installs, you need to manage your shell PATH. In particular, you should ensure that the framework bin directory of the Python(s) you want to use are on your PATH. So for MacPorts, something like this in ~/.bash_profile:

export PATH="/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin:/opt/local/bin:$PATH

or for the python.org installers:

export PATH="/Library/Frameworks/Python.framework/Versions/2.6/bin:$PATH

In addition, generally you need to install a separate version of your favorite Python package management add-ons (like pip or the easy_install command from Distribute or setuptools) for each Python instance you use. Each one is installed into the default script directory (i.e. the framework bin directory). MacPorts takes care of that automatically when you install its version of the package (sudo port install py26-pip or sudo port install py31-distribute). Note that Apple supplies a set of easy_install links in /usr/bin for its Pythons and they install packages to /Library/Python. You need to make sure you use the easy_install or pip appropriate for your instance of Python. Making sure the instance's framework bin directory is first in the PATH ensures that.

Note, if you use MacPorts you can use its python_select command to manage which Python instance is pointed to by /opt/local/bin/python. Then if /opt/local/bin appears earlier in your PATH list than other directories with a python it will govern which is your default python. But you really need to manage the PATH by adding the framework bin directory so python_select is generally not all that useful.

Another positive aspect of using MacPorts (or similar broad-based distributor) is that it makes available many popular Python packages including handling the occasional messy details of building and linking to 3rd-party C libraries not shipped with Mac OS X. For example, the MySQLdb and PIL packages are frequent stumbling blocks on Mac OS X because of the MySQL client libraries and libjpeg, but MacPorts makes it much easier to get everything right. But you do have the choice to install things directly using MacPorts (sudo port py26-pil) or via pip (sudo pip install pil) or easy_install, or you can download the source, and do it yourself (sudo python setup.py ...). And because all of them under the covers use Python's Distutils to do the dirty work, all of these methods will generally end up installing a package's files in the same place for a given Python instance.

This is an old question, but I have struggled in the past with upgrading Python and virtual environments. I have figured out how to get this to work for the stuff I do. Hopefully this will help someone.

Setup:
Mac running 10.14
Python 3.7
Various packages: Pandas, IPython, Paramiko, etc.

Python:
Just use the installer they provide at Python.org. Click on the "Download" button for macOS, download the packages and click through the installer. Done. When it comes time to upgrade to the next Python 3 version, repeat the process.
To get the Python3 interpreter on the command-line instead of the Python2 one, use python3 instead of python.

Getting Rid of Older Python3 versions
If you don't like the idea of having old Python versions hanging around, open the Applications folder and drag-and-drop the old Python3 version you don't want to the trash. DO NOT do this to the Python2 versions unless you know what you a re doing. Some of these came with the OS and will likely cause trouble if they disappear.

Installing Packages:.
@Ned Deily's suggestion to use virtual environments is a good one if you want to make installing packages easier. The virtual environment tool for Python2 is 'virtualenv'. That tool is called 'venv' in Python3.

Step 1:
Create a directory to hold all of your virtual environments. This isn't necessary, but this is how I keep track of mine. I keep two separate dirs named 'virtenvs' and 'venvs' depending on whether I'm installing a package for Python2 or Python3. For this example, we'll use Python3:

cd ~
mkdir venvs
cd venvs

Now, I name my packages after my project or after the major package I've installed. For this example, let's just call it 'project1':
venv project1

Step 2:
"Activate" the virtual environment using the following command. You'll need to to this every time you start a new terminal or session.
source ~/venvs/project1/bin/activate

Pip changes a lot, so update it
pip install --upgrade pip

Step 3:
Now you're ready to start installing packages into your new virtual environment:
pip install pandas
etc.

When you're done or you want to switch to a different virtual environment, just type:
deactivate

What's Next:
Explore the web for more info on Python virtual environments. Using this approach, I've not run into a package that required MacPorts, Anaconda, etc. There are undoubtedly packages that do. YMMV. Hopefully this approach will work for you.

I just used 'pythonbrew', which I easy_installed with the stock system Python on 10.6.7. I had to use the '--force' option to bypass the post-compile tests though, as I had failures with test_dl and test_readline (a quick google check did not show these to be a real issue).

$ sudo easy_install pythonbrew

Once installed, run this for more instructions:

$ pythonbrew_install

So I now have a nicely isolated python-2.7.1 installation without interfering with the system installation.

I would recommend using Homebrew, as do the Python Guide people:

Here is a fantastic step by step guide to getting started:

http://docs.python-guide.org/en/latest/starting/install/osx/#install-osx