Calling os.fsync on the stdout/stderr file descriptors kills a subprocess

After spawning a subprocess using the Python subprocess library, I'm using stderr to pass a message from the child process to the parent process containing some serialized data. I then want the parent to return (via stdin) the result o开发者_如何学Cf a function applied to this data.

In essence, I have a function inside the subprocess that does something like this:

sys.stderr.write("some stuff to write")
# some time later
some_var = sys.stdin.read()

However, this completes locks the parent while waiting for the stderr input, so I tried to call:

sys.stderr.flush()
os.fsync(sys.stderr.fileno())

However, this doesn't work. Nothing after the os.fsync is executed. In addition, when I call proc.poll() in the parent process, it appears, the child's return code is 1.

What can I do to prevent this? Should I consider another approach?

I would consider another approach. You may use an indipendent process (multiprocessing.Process) and using two queues to communicate with it (multiprocessing.Queue) one for the input and the other one for the output. Example on starting the process:

import multiprocessing

def processWorker(input, result):
    work = input.get()
    print work
    result.put(work*work)

input  = multiprocessing.Queue()
result = multiprocessing.Queue()

p = multiprocessing.Process(target = processWorker, args = (input, result))
p.start()

input.put(2)
res = result.get(block = True)
print res

Then you may iterate passing again it. Usage of multiprocessing.Queue is more robust since you do not need to rely on stdout/err parsing and you also avoid related limitation. Moreover you can easily manage more subprocesses.

Then, you can also set a timeout on how long you want a get call to wait at max, eg:

import queue
try:
    res = result.get(block = True, timeout = 10)
except Queue.Empty:
    print error

Here is how you could prevent I/O deadlock without radically changing the approach. In the child:

import os, fcntl
map(lambda f: fcntl.fcntl(f.fileno(), fcntl.F_SETFL, os.O_NONBLOCK),
    (sys.stdin, sys.stdout, sys.stderr))

In the parent, do the same for the pipe file descriptors to/from the child. Then when you use select.select() to communicate you will no longer have these lock-ups. However, you must use select() even before writing, you can receive EAGAIN errors when you try to read/write, and depending on your application logic you may have an indefinite wait situation anyway.

I honestly recommend looking into the Twisted framework, which has child process functionality built in: http://twistedmatrix.com/documents/current/core/howto/process.html