While this question has a python backend, the question is not tied to python itself, but rather about extension mechanisms and how to register/lookup for plugins.

In python, the concept of entrypoints was introduced by setuptools, and is tied to the metadata of installed python distributions (called packages in other packaging systems).

As of my understanding, one of the features provided by entrypoints is to allows an application to define a place where others can put things in, so any application wanting to use an entrypoint can get a list of registered classes/functions there. Let's take an example:

• Foo defines the entrypoint "entrypoint1" and look for plugins registered under this name.
• Bar register a callable (Bar.callable) on the "entrypoint1" entrypoint.
• Any python script is then able to list Bar.callable as one of the registered callables for "entrypoint1".

With setuptools, applications registers entrypoints at install time and the information is stored in metadata related to packaging, called .egginfo (which usually contain information about the distribution name, its dependencies, and some more metadata about packaging).

I have the feeling that packaging metadata is not the right place to store this kind of information, as I don't get why this information is tied to packaging.

I'm curious to hear about such entrypoints/extensions/plugins features in other languages, and especially if the concept is tied to metadata and packaging or not. And so the question is…

Do you have examples I should look at? Could you explain why the design choices have been made that way?

C开发者_StackOverflow中文版an you see different ways to handle this problem? Do you know how this problem has already been solved in different tools? What are the downsides and the advantages of the current python implementation over others?

What I've found so far

I've found in different projects, a way to create and to distribute "plugins", which are especially taking care about the "how do we make plugins".

For instance, libpeas (the gobject plugin framework) defines a set of ways to extend a default behavior by specifying plugins. While this is interesting, I am just interested by the "registering and finding" (and eventually loading) part of it.

Here are some of my findings so far:

Libpeas defines its own metadata file (*.plugin), which stores information about the type of the callable (it is possible to have different plugins in different languages). The main information here is the name of the module to load.

Maven have a design document containing information on how stuff is managed there. Maven manages plugins with their dependencies and metadata so it seems like an interesting place to look for how they implemented things.

As specified in their documentation, maven plugins are using annotations (@goal) on the classes, which is then used to find all the plugins registered with a particular @goal. While this approach is possible in static languages, it is not in interpreted languages as we only know what are all the possible classes / callables at one given point of the time, which may change.

Mercurial uses a central configuration file (~/.hgrc), containing a mapping of the name of the plugin to the path it can be found.

Some more thoughts

While that's not an answer to this question, it is also interesting to note how setuptools entrypoints are implemented, and how they compare in term of performance, with the mercurial ones.

When you ask for a particular entry point with setuptools, all the metadata are read at run time and a list is built that way. This means this reading can take some time if you have a lot of python distributions on your path. Mercurial on the other side hardcode this information into a single file, which means you have to specify the complete path to your callable there, then registered callables are not "discovered" but "read" directly from the configuration file. This allows finer grained configuration on what's should be available and what should not be and seems faster.

On the other side, as the python path can be changed at runtime, this means that callables provided on such a way would have to be checked against the path in order to know if they should be returned or not in all situations.

Why entrypoints are currently tied to packaging

It is also interesting to understand why entrypoints are tied to packaging in setuptools. The main reason is that it seems useful that python distributions can register part of themselves as extending an entrypoint at installation time: then installing means also registering the entrypoints: no need for an extra registration step.

While this works fairly well in a large majority of cases (when python distributions are actually being installed) it doesn't when they are not installed or simply not packaged. In other words, from what I understand, you can't register an entrypoint at runtime, without having an .egg-info file.

as one of possible ideas, you can look at OSGi concept, that is used for plugin system management of Eclipse. It could be overkill for your specific case, but definitely the source of inspiration.

I'd like to answer the question on "How to register a python entry point without packaging nor installing a module". I think it is implied from your thoughts and findings and is what actually drove me mad for a long time.

As to the rest, albeit a very interesting topic, I think the question is too broad to be entirely answered (I am actually surprised it is not closed due to broadness) and it has been hard for me to find it until I have already been able to answer this part myself.

How to register a python entry point without packaging nor installing a module

For the common practitioners of laying python modules or bare .py files in an importable path, usually ./src, ./lib or the flat working directory, there is no need to package anything in order to register an entry point.

With your example, it is enough to drop a file in the same importable directory under .../bar.dist-info/entry_points.txt with the following content

# bar.dist-info/entry_points.txt
[entrypoint1]
bar = Bar.callable

so that the arbitrary callables can be retrieved by

import importlib.metadata
importlib.metadata.entry_points()["entrypoint1"]

This way everything is actually independent from setuptools nor any other packaging tooling.

This is fairly undocumented and I only found it after painfully debugging the importation of setuptools' pkg_resources. I have not digged any further about the placement requirements for the entry_points.txt but it seems that any arbitrary *.{egg,dist}-info/ folder does cut it as long as it is in the python path.

Relating this to your thoughts, since the python importable paths can be tampered via PYTHONPATH, PYTHONHOME, PYTHONUSERSITE and other site internal features, I think this behavior is very convenient so as to only register entry points that are within reach of Python's import machinery. Better than crafting a single file whose records would need to be runtime-checked, IMHO.

Alternatives?

Further reading on https://docs.python.org/3/library/importlib.html#setting-up-an-importer gives me the idea that it should be possible to register entry points via importlib.machinery entry finders but I haven't played with that. It might be the only way to achieve it if you are loading modules from crazy non-file sources like https://github.com/nvbn/import_from_github_com

Exemplar application

pytest lets you register plugins via the "pytest11" entry point identifier. This is how plugin writers are expected to announce themselves so that pytest --fixtures will list their packaged fixtures, for example.

Yet, as a pytest end user you are not expected to register any hooks, fixtures nor anything via that entry point! They will be loaded from your local documented conftest.py files. And if developing actual local plugins, you can register them with a PYTEST_PLUGINS environment variable.

So pytest "the application" offers you equivalent mechanisms to register aribtrary functionality whether you are a plugin developer, a plugin user or simply writing your local hacks. Entry points are only the registration mechanism for "properly installed" plugins.

I believe this should be the way to go for any applications needing to register plugins.

As you started out speaking about programming language implementation / processing, it may be worth noting that recently gcc acquired plugin capabilities too.