Any way to execute a piped command in Python using subprocess module, without using shell=True?

I want to run a piped command line linux/bash command from Python, which first tars files, and then splits the tar file. The command would look like something this in bash:

> tar -cvf - path_to_archive/* | split -b 20m -d -a 5 - "archive.tar.split"

I know that I could execute it using subprocess, by settings shell=True, and submitting the whole command as a string, like so:

import subprocess    

subprocess.call("tar -cvf - path_to_archive/* | split -b 20m -d -a 5 - 'archive.tar.split'", shell=True)

...but for security reasons I would like to find a way to skip the "shell=True" part, (which takes a list of strings rather than a full command line string, and which can not handle the pipe char correctly). Is开发者_开发技巧 there any solution for this in Python? I.e., is it possible to set up linked pipes somehow, or some other solution?

If you want to avoid using shell=True, you can manually use subprocess pipes.

from subprocess import Popen, PIPE
p1 = Popen(["tar", "-cvf", "-", "path_to_archive"], stdout=PIPE)
p2 = Popen(["split", "-b", "20m", "-d", "-a", "5", "-", "'archive.tar.split'"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Note that if you do not use the shell, you will not have access to expansion of globbing characters like *. Instead you can use the glob module.

tar can split itself:

tar -L 1000000 -F name-script.sh cf split.tar largefile1 largefile2 ...

name-script.sh

#!/bin/bash
echo "${TAR_ARCHIVE/_part*.tar/}"_part"${TAR_VOLUME}".tar >&"${TAR_FD}"

To re-assemble

tar -M -F name-script.sh cf split.tar

Add this to your python program.

Is there any reason you can't use tarfile? | http://docs.python.org/library/tarfile.html

import tarfile
tar = tarfile.open("sample.tar.gz")
tar.extractall()
tar.close()

Just write like a file like object using tarfile rather than invoking subprocess.

Shameless plug, I wrote a subprocess wrapper for easier command piping in python: https://github.com/houqp/shell.py

Example:

shell.ex("tar -cvf - path_to_archive") | "split -b 20m -d -a 5 - 'archive.tar.split'"