Python's subprocess module returning different results from Unix shell

I'm trying to get a list of the CSV files in a directory with python. This is really easy within unix:

ls -l *.csv

And, predict开发者_JAVA技巧ably, I get a list of the files that end with .csv in my directory. However, when I attempt the Python equivalent using the Subprocess module:

>>> import subprocess as sp
>>> sp.Popen(["ls", "-l", "*.csv"], stdout = sp.PIPE)
<subprocess.Popen object at 0xb780e90c>
>>> ls: cannot access *.csv: No such file or directory

Can somebody please explain what's going on?

Edit: Adding shell = True removes the error, but instead of getting a list of just CSV files, I get a list of all the files in the directory.

If you want it to behave as it does at the shell, you need to pass shell=True (your mileage may vary here, depending on your system and shell). In your case the problem is that when you do ls -l *.csv, the shell is evaluating what * means, not ls. (ls is merely formatting your results, but the shell has done the heavy lifting to determine what files match *.csv). Subprocess makes ls treat *.csv literally, and look for a file with that specific name, which of course there aren't any (since that's a pretty hard filename to create).

What you really should be doing is using os.listdir and filtering the names yourself.

Why not use glob instead? It's going to be faster than "shelling out"!

import glob
glob.glob('*.csv')

This gives you just the names, not all the extra info ls -l supplies, though you can get extra info with os.stat calls on files of interest.

If you really must use ls -l, I think you want to pass it as a string for the shell to do the needed star-expansion:

proc = sp.Popen('ls -l *.csv', shell=True, stdout=sp.PIPE)

When you enter ls -l *.csv at the shell, the shell itself expands *.csv into a list of all the filenames it matches. So the arguments to ls will actually be something more like ls -l spam.txt eggs.txt ham.py

The ls command doesn't understand wildcards itself. So when you pass the argument *.csv to it it tries to treat it as a filename, and there is no file with that name. As Nick says, you can use the shell=True parameter to have Python invoke a shell to run the subprocess, and the shell will expand the wildcards for you.

p=subprocess.Popen(["ls", "-l", "*.out"], stdout = subprocess.PIPE, shell=True)

causes

/bin/sh -c ls -l *.out

to be executed.

If you try this command in a directory, you'll see -- in typical mystifying-shell fashion -- all files are listed. And the -l flag is ignored as well. That's a clue.

You see, the -c flag is picking up only the ls. The rest of the arguments are being eaten up by /bin/sh, not by ls.

To get this command to work right at the terminal, you have to type

/bin/sh -c "ls -l *.out"

Now /bin/sh sees the full command "ls -l *.out" as the argument to the -c flag.

So to get this to work out right using subprocess.Popen, you are best off just passing the command as a single string

p=subprocess.Popen("ls -l *.out", stdout = subprocess.PIPE, shell=True)
output,error=p.communicate()
print(output)