Run pika ioloop in background or use custom ioloop

I have a feeling that this should really not be all that difficult, yet I have had little success so far.

Say I have a class called PikaClass that wraps pika and provides some business methods.

def PikaClass(object):
  def __init__(self):
     # connect to the broker
     self.connection = pika.SelectConnection(<connection parameters>, self.on_connect)
     # ..other init stuff..

  def on_connect(self, connection):
     # called when the connection has been established 
     # ..open a channel, declare some queues, etc.

  def start(self):
     # start the polling loop 
     self.connection.ioloop.start()

  def foo(self, **kwargs):
     # do some business logic, e.g., send messages to particular queues

Intuitively, this is what I would like to achieve: a user creates an instance of PikaClass, sets the loop going in the background, and then interacts with the object by calling certain business methods

p = PikaClass()
p.start()
bar = p.foo(..)

The problem is that p.start() blocks and prevents the main code from interacting with the object once start() has been called. My first thought was to wrap the call in a thread:

Thread(target=p.start()).start()
bar = p.foo(..)

But that still blocks and you never get to p.foo(..). The docs mention that you shouldn't share a connection between threads so that may cause a problem somewhere.

I have also tried using AsyncoreConnection instead of SelectConnection, and cal开发者_JAVA百科ling _connect() directly (instead of using the ioloop) but that does not have any effect (nothing happens).

So how can I run the ioloop in the background, or at least run my own ioloop?

Note: This is Python 2.6 on win64 (xp) with the latest pika 0.9.4

You are calling 'p.start' instead of passing it as a parameter. The code should be:

Thread(target=p.start).start()

Thread will call p.start when Thread.start is executed.

I'm not sure if this will solve your problem, but it may help you reach the solution.

The GIL is not a problem here, because the ioloop spends nearly all of its time in the select(2) system call, during which time the GIL is released and other Python threads can run off and perform other tasks.

The simplest approach would be to set up and tear down a queue connection for every request. You might think this would be too expensive — since it requires re-authentication and (possibly) repeat SSL negotiation with every connection — but it should be the simplest and most robust and easiest to write, which should be your controlling factors unless you know that setup and teardown will in fact hurt performance in your application overall (which is best measured by testing).

Another approach would be to only start() the ioloop once you had a message to send, and have the method that receives the reply stop the ioloop so that your program gets control again. You can make the ioloop return early with:

    connection.ioloop.poller.open = False

and then remembering to set it back to True before you call start() again to wait for another reply.

You might need a second process, I don't know much about pika but if it's pure-python then it'll want to keep the GIL - remember you can only execute one thread at a time per process, no matter how many cores you have, due to limitations in the reference-counter used for python's wonderful garbage-collection.

If you start a new process before you do /anything/ else in your if __name__ == "__main__": block, and set it off waiting for events on a loop, you can send your second process the instruction to execute a piece of work then wait for it to send the result back to you. You'll probably want to implement an indexing system of some sort so you can fire your work to the other process and forget about it until you need the answer.

My only advice if you go down this bath is to avoid side-effects or you'll have to consider race conditions, deadlocks, and all the other horrible stuff that the GIL protects you from. Send your process the information it needs before it starts, or make sure you don't modify it before it asks you for that piece of data.

This is one of those few occasions where Python will hand you the gun with only a polite note saying not to aim it at your foot.