What are prevalent techniques for enabling user code extensions in Python?

I'm looking for techniques that allow users to override modules in an application or extend an application with new modules.

Imagine an application called pydraw. It开发者_JAVA百科 currently provides a Circle class, which inherits Shape. The package tree might look like:

/usr/lib/python/
└── pydraw
    ├── __init__.py
    ├── shape.py
    └── shapes
        ├── circle.py
        └── __init__.py

Now suppose I'd like to enable dynamic discovery and loading of user modules that implement a new shape, or perhaps even the Shape class itself. It seems most straightforward for a user's tree to have the same structure as the application tree, such as:

/home/someuser/python/
└── pydraw
    ├── __init__.py
    ├── shape.py       <-- new superclass
    └── shapes
        ├── __init__.py
        └── square.py   <-- new user class

In other words, I'd like to overlay and mask an application tree with same-named files from the user's tree, or at least get that apparent structure from a Python point of view.

Then, by configuring sys.path or PYTHONPATH, pydraw.shapes.square might be discoverable. However, Python's module path search doesn't find modules such as square.py. I presume this is because __method__ already contains a parent module at another path.

How would you accomplish this task with Python?

Discovery of extensions can be a bit brittle and complex, and also requires you to look through all of PYTHONPATH which can be very big.

Instead, have a configuration file that lists the plugins that should be loaded. This can be done by listing them as module names, and also requiring that they are located on the PYTHONPATH, or by simply listing the full paths.

If you want per user level configurations, I'd have both a global configuration file that lists modules, and a per user one, and just read these config files instead of trying some discovery mechanism.

Also you are trying to not only add plugins, but override components of your application. For that I would use the Zope Component Architecture. It is however not yet fully ported to Python 3, but it's designed to be used in exactly this kinds of cases, although from your description this seems to be a simple case, and the ZCA might be overkill. But look at it anyway.

• http://www.muthukadan.net/docs/zca.html
• http://pypi.python.org/pypi/zope.component

If you want to load python code dynamically from different locations, you can extend the search __path__ attributes by using the pkgutil module:

By placing these lines into each pydraw/__init__.py and pydraw/shapes/__init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

You will be able to write import statement as if you had a unique package:

>>> import pydraw.shapes
>>> pydraw.shapes.__path__
['/usr/lib/python/pydraw/shapes', '/home/someuser/python/pydraw/shapes']
>>> from pydraw.shapes import circle, square
>>>

You may think about auto-registration of your plugins. You can still use basic python code for that by setting a module variable (which will act as a kind of singleton pattern).

Add the last line in every pydraw/shapes/__init__.py file:

 from pkgutil import extend_path
 __path__ = extend_path(__path__, __name__)

 # your shape registry
 __shapes__ = []

You can now register a shape in top of its related module (circle.py or square.py here).

 from pydraw.shapes import __shapes__
 __shapes__.append(__name__)

Last check:

 >>> from pydraw.shapes import circle,square
 >>> from pydraw.shapes import circle,square,__shapes__
 >>> __shapes__
 ['pydraw.shapes.circle', 'pydraw.shapes.square']

A method I used to handle such a problem was with a provider pattern.

In your module shape.py :

class BaseShape:
   def __init__(self):
      pass
provider("BaseShape", BaseShape)

and in the user's shape.py module :

class UserBaseShape:
   def __init__(self):
      pass
provider("BaseShape", UserBaseShape)

with provider method doing something like this :

def provider(provide_key, provider_class):
   global_providers[provide_key] = provider_class

And when you need to instanciate an object, use a ProvideFactory like this

class ProvideFactory:
   def get(self, provide_key, *args, **kwargs):
      return global_providers[provide_key](*args, **kwargs)

You can detect file changes within a certain directory with OS-specific methods. For all operating systems, there exist file monitoring tools that produce events when a file or directory is changed. Alternatively, you can continuously search for files newer than the time of the last search. There are multiple possible solutions, but in any case:

• configuring a plugins directory makes things a lot easier than monitoring your complete file system.
• looking for file changes in a separate thread is probably the best solution
• if a new .py file is found, you can import it using the __import__ built-in function
• if a .py is changed, you can re-import it with the reload built-in function
• If a class is changed, the instances of that class will still behave like instances of the old class, so make sure to recreate instances when necessary

EDIT:

If you add your plugins directory as the first directory in PYTHONPATH, that directory will have priority over the other directories in pythonpath, e.g.

import sys
sys.path.insert(0, 'PLUGIN_DIR')