Appending lists from files to a single list in Python

I'm trying to write a function that reads files from a "deferred" directory which contains files that contain lists. Here's what the files in the deferred folder contain:

'173378981', '45000', '343434', '3453453', '34534545', '3452342', '234234', '42063008', 'Exempted', '10000'
'1000014833', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'1000009598', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'279483421', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'1000009600', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'389453080', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'
'1000009602', '0', '0', '0', '0', '0', '0', '0', 'Exempted', '0'

The function used to write the file(s):

def storeDeferredRecords(records):
    """docstring for createFile"""
    now = datetime.datetime.now()
    filename = deferredDir + '/' + now.strftime("%Y%m%d-%H%M%S")
    f = open(filename, 'w')
    newlist = map(lambda(x): str(x)[1:-1], records)
    for item in newlist:
        f.write("%s\n" % item)
    f.close

I need help with the function used to read the file. I was only able to write this:

def getDeferredRecords():
        """docstring for getDeferredRecords"""
        infiles = [infile for infile in glob.glob(deferredDir + '/*')]
                <code to read the contents of each file here>

Can someone help me out? I need to read the lines and insert them i开发者_运维技巧nto a list. This list will then be merged with records from separate CSV file.

First, the last line in the store function needs to be like this f.close()

Your store function saves the values in a newline-separated manner. To read all the files, should be enough:

def getDeferredRecords():
    """docstring for getDeferredRecords"""
    return dict((infile, list(iter(file(infile)))) 
                     for infile in glob.glob(deferredDir + '/*'))

Explanation: a file is iterable, so you can do for line in file: print line for example. With list(iter(file)) you have the lines of a file in a list. dict((a, b) for a, b in foo) returns a dictionary with {a: b} pairs. The return value of the function is a dictionary with the format {filename: list_of_lines_in_file}. Keep in mind that the list elements are strings with a trailing newline.

See the csv module:

BigList = []
for filename in glob.glob(deferredDir + '/*'):
    PartList = csv.reader(open(filename))
    BigList.extend(PartList)

Is that what you had in mind?

The Python cvs module is likely a good answer:
http://docs.python.org/library/csv.html

Question:

glob.glob() returns an iterable already, so I do not see the point here...

[infile for infile in glob.glob(deferredDir + '/*')]

Rather:

BigList = []
for filename in glob.glob(deferredDir + '/*'):
    #CVS read code here
    #add to BigList

Food for thought.

Incorporating ideas from Tim Pietzcker, here are the re-written functions:

def storeDeferredRecords(records):
    """docstring for createFile"""
    now = datetime.datetime.now()
    filename = deferredDir + '/' + now.strftime("%Y%m%d-%H%M%S")
    f = csv.writer(open(filename, 'w'), delimiter=',')
    f.writerows(records)

def getDeferredRecords():
    """docstring for getDeferredRecords"""
    for filename in glob.glob(deferredDir + '/*'):
        def_records = csv.reader(open(filename,'r'))
        records.extend(def_records)

I used csv.writer instead of using the previous code block:

f = open(filename, 'w')
newlist = map(lambda(x): str(x)[1:-1], records)
for item in newlist:
        f.write("%s\n" % item)
f.close

Thanks to all those who replied!