Splitting 25mb .txt file into smaller files using text delimiter
Regards, SO
I am new to python and Perl. I have been trying to solve a simple problem and getting tied in knots with syntax. I hope someone has the time and patience to help. I have a 25mb file in ".txt" format which contains news-wire articles going back to 1970. Each news story is concatenated to the next, with only the "Copyright" statement to delimit. Each news story starts with "Item XX of XXX DOCUMENTS". There are certain metadata that are repeated throughout, I will use these for tagging later on.
I wish to split this 25mb file into separate .txt files, each containing one news story (i.e. the text between "DOCUMENTS" and "Copyright", saving each with a different name (obviously).
I am trying to 1 ) open the file... 2) iterate over lines in the file checking for the eof delimiter, and if it is not present writing the line to a list 3)write that list to a seperate small file.
I'm having big problems with changing filenames using the counter, and how do I make Python start from where I left off, is the "seek" function appropriate?
so far I have been trying this approach, completely unsuccessfully:
myfile = open ("myfile.txt", 'r')
filenumber = 0
for line in myfile.readline():
filenumber += 1
w=0
while myfile.readline() != '\s+DOCUMENTS\s*\n'
### read my line into a list
mysmallfile()['w'] 开发者_StackOverflow= [myfile.readline()]
w += 1
output = open('C:\\Users\\dunner7\\Documents\###how do I change the filename each iteration???', 'w')
output.writelines(mysmallfile)
###go back to start.
Thank you for your time and patience.
RD
Here is a sample of the text file:
1 of 575 DOCUMENTS
The Washington Daybook
January 28, 2011
Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.
TIME: 8 a.m.
LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md.
CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]
LOAD-DATE: November 28, 2010
LANGUAGE: ENGLISH
TYPE: Meeting
Copyright 2011 Federal Information and News Dispatch, Inc.
2 of 575 DOCUMENTS
The Washington Daybook
January 27, 2011
Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.
TIME: 8 a.m.
LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md.
CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]
LOAD-DATE: November 28, 2010
LANGUAGE: ENGLISH
TYPE: Meeting
Copyright 2011 Federal Information and News Dispatch, Inc.
3 of 575 DOCUMENTS
FNS DAYBOOK
January 12, 2011 Wednesday
FUTURE EVENTS
EVENT: MEETING - HEALTH AND HUMAN SERVICES DEPARTMENT (HHS); FOOD AND DRUG ADMINISTRATION (FDA) (F.R. PAGE 72832); LOCATION: Hilton Washington DC North/Gaithersburg, Ballroom, 620 Perry Parkway, Gaithersburg, Md. -- January 27, 2011 8:00 am
SECTION: FEDERAL AGENCIES AND DEPARTMENTS - FUTURES
LENGTH: 72 words
SUBJECT: Health and Human Services Department (HHS); Food and Drug Administration (FDA) (F.R. Page 72832) holds a meeting of the Neurological Devices Panel of the Medical Devices Advisory Committee to discuss and make recommendations regarding the possible reclassification of devices indicated for use in electroconvulsive therapy, January 27-28.
CONTACT: James Engles, 800-741-8138 [Note: Use the code: 3014512513, when calling for information.]
LOAD-DATE: January 10, 2011
LANGUAGE: ENGLISH
PUBLICATION-TYPE: Event Schedules
Copyright 2011 Federal News Service
All Rights Reserved
Something like that:
filenumber = 0
outfile = None
with open('source_file.txt') as f:
for line in f:
if line.strip() == 'DOCUMENTS':
filenumber += 1
outfile = open('result%03d.txt' % filenumber, 'w')
elif line.strip().startswith('Copyright') and outfile:
outfile.close()
outfile = None
elif outfile:
outfile.write(line)
if outfile:
outfile.close()
I had to guess lots of things because I don't know exacly how the file looks like. Post the file if you have further problems.
Several problems along the way:
- You're using
myfile.readline()
in both the loop iterator and when writing to a small file -- just useline
when you want to know the contents of the current line - The
myfile.readline() != 'string'
bit is always going to fail, this isn't actually a regexp match. mysmallfile()['w']
looks overly clever (if you're using themysmallfile()
function to return an array or dictionary indexed with 'w' to retrieve an L-value(!)) or like you're just way off course here :)- You open a file, write a line into it, but then never close the file. This is recipe for failing after 1000 lines when you run out of open file descriptors.
There's a lot of cleanup that went into nosklo's suggestion, I hope my post can help you understand how he got to his. :)
精彩评论