Getting progress message from a subprocess

I want to start a program which needs several minutes to complete. During this time I want to read the progress message of the program (which are printed on the stdout). The problem is that I cannot find a way to read out its output during its run.

The only function I found to read out the output of a program is Popen.communicate(), but this method waits until the process finishes. So it is impossible to get the progress and make it visible to the user in a special formatted way.

Is it possible to do this another way?

When I run the process with subproce开发者_如何学Pythonss.popen with my script I see the output of the program on the screen. Is it possible to hide it? (Ubuntu 10.10, normal terminal)

Simplest is to call Popen with the keyword argument stdout=subprocess.PIPE.

p = subprocess.Popen(["ls"], stdout=subprocess.PIPE)
while True:
    line = p.stdout.readline()
    if not line:
        break
    print line

To see this in action, here are two sample scripts. Make them both in the same directory and run python superprint.py

printandwait.py:

import time
import sys
print 10
sys.stdout.flush()
time.sleep(10)
print 20
sys.stdout.flush()

superprint.py:

import subprocess
import sys
p = subprocess.Popen(["python printandwait.py"], shell=True, stdout=subprocess.PIPE)
while True:
    print "Looping"
    line = p.stdout.readline()
    if not line:
        break
    print line.strip()
    sys.stdout.flush()

You can do a poll on the status of your subprocess and keep outputting lines.

p = subprocess.Popen('ls;sleep 10', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

rc = p.poll()
while rc != 0:
    while True:
        line = p.stdout.readline()
        if not line:
            break
        print line
    rc = p.poll()

assert rc == 0

It's certainly possible: my package python-gnupg does exactly this, spawning gpg (Gnu Privacy Guard) under a subprocess. In the general case you need to specify subprocess.PIPE for the subprocess stdout and stderr; then create two separate threads which read the subprocess stdout and stderr to wherever you like.

In the case of python-gnupg, status messages from gpg are read and acted upon while the gpg process is running (not waiting until it's finished).

Basically, pseudocode is

process = subprocess.Popen(..., stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr = process.stderr
rr = threading.Thread(target=response_reader_func, args=(process.stderr,))
rr.setDaemon(True)
rr.start()

dr = threading.Thread(target=data_reader_func, args=(process.stdout,))
dr.setDaemon(True)
dr.start()

dr.join()
rr.join()
process.wait()

The reader functions are typically methods of an enclosing class which do the right thing based on what they're reading (in your case, updating progress info in some way).