pthread with callback to python VM

Let's say I have a python script which loads a shared library (SL) through ctypes.

• The SL sets up a pthread T1
• The python script configures callbacks through the SL i.e. python script calls functions from the SL with references to python callables

alt text http://www.gliffy.com/pubdoc/1993061/L.jpg

Now, let's say T1 calls a "callback" function, are the following assumptions true:

• the callback function on the Python side is executed within the context of T1
• I can use a queue to communicate between T1 and the Python VM
• I would need to poll the said queue on the Python VM side

I understand all the concepts of threading, shared state etc. but I haven't dug very deep on the mul开发者_StackOverflow中文版ti-threading side with Python. Since there is an adaptation layer which I do not know enough at the moment (ctypes), I am afraid I'll be missing some key aspects of the process.

Polling the queue isn't normally necessary (you can devote another thread on the Python side to doing blocking .get calls on it), but that's not a big deal. Problem is, with such an arrangement, you might get caught by the GIL -- see the three links from this wikipedia page for ample treatments thereof.

When you interface to/from C with C (or Cython) code using the Python C API, you can release and acquire the GIL pretty simply, at least, hopefully avoiding deadlocks and the like; with ctypes, GIL operations are automated in callback to/from C situations, so if there's any other lock in play a deadlock is a risk (since things are not within your control you can't easily ensure Djikstra's Banker Algorithm is applied).

This can be easily implemented with Cython . I have written a sample at https://github.com/sachinpc/cython/tree/master/Demos/mt_callback

The example shows how to wrap python callback in cython, when being called from a pthread. For your specific use-case , you could add to a Queue or dequeue object in the thread code from the callback function . In the sameple , I have added prints for both pthread-id and python-thread-id in the extension .