incremental output with subprocess.PIPE

I am using subprocess module for running commands from another app I know that you are able to do the following

import subprocess

app = subprocess(args, stdout=subprocess.PIPE)
out, err = app.communicate()
print out

I would like the outpu开发者_如何学JAVAt to be displayed as it happening as supposed to one big blob at the end. Ideas?

The problem may be that the command you are running in the subprocess buffers its output: in which case, in Blender's answer, the process.stdout.read(1) would not return until the output buffer of the subprocess filled up, causing it to be flushed, and thus seen by the parent process.

See this answer and this one for more information about how buffering affects things.

Here's some scripts to illustrate:

# spew.py
import sys
import time

for i in range(10):
    print 'Message no. %d' % i
    #sys.stdout.flush()
    time.sleep(1)

and

# runspew.py
import subprocess
import time

start = time.time()
p = subprocess.Popen(['python', 'spew.py'], stdout=subprocess.PIPE)
data = p.stdout.read(1)
print 'Elapsed time to first byte read: %.2f' % (time.time() - start)
data = p.stdout.read()
print 'Elapsed time to last byte read: %.2f' % (time.time() - start)
p.wait()

Now, put both scripts in the same directory and run python runspew.py. You'll see something like this:

Elapsed time to first byte read: 10.05
Elapsed time to last byte read: 10.05

Now, remove the comment from sys.stdout.flush() in spew.py and do python runspew.py again. You'll now see something like this:

Elapsed time to first byte read: 0.02
Elapsed time to last byte read: 10.04

Notice that runspew.py didn't change: only the program which was run as the subprocess did. If the program you're running can't be made to run unbuffered (or flush frequently), then you'll have to use expect/unbuffer as described in the other answers I linked to.

I made a Python shell implementation once, so here's the code I used to run commands. It ran wget and nano fine, so I think the code would suit your needs:

  process = subprocess.Popen(shlex.split(command), stdout = subprocess.PIPE, stderr = subprocess.STDOUT)

  while True:
    output = process.stdout.read(1)

    if output == '' and process.poll() != None:
      break

    if output != '':
      sys.stdout.write(output)
      sys.stdout.flush()

Basically, you have to write directly into stdout.