Sending large file to PIPE input in python

I have the following code:

sourcefile = open(filein, "r")
targetfile = open(pathout, "w")

content= sourcefile.read():

p = Popen([SCRIPT], stdout=targetfile, stdin=PIPE)
p.communicate(content)

sourcefile.close()
targetfile.close()

The data in sourcefile is quite large, so it takes a lot of memory/swap to store it in 'content'. I tried to send the file directly to stdin with stdin=sourcefile, which 开发者_JAVA百科works except the external script 'hangs', ie: keeps waiting for an EOF. This might be a bug in the external script, but that is out of my control for now..

Any advice on how to send the large file to my external script?

Replace the p.communicate(content) with a loop which reads from the sourcefile, and writes to p.stdin in blocks. When sourcefile is EOF, make sure to close p.stdin.

sourcefile = open(filein, "r")
targetfile = open(pathout, "w")

p = Popen([SCRIPT], stdout=targetfile, stdin=PIPE)
while True:
    data = sourcefile.read(1024)
    if len(data) == 0:
        break
    p.stdin.write(data)
sourcefile.close()
p.stdin.close()

p.wait()
targetfile.close()