Help parsing text file in python

Really been struggling with this one for some time now, i have many text files with a specific format from which i need to extract all the data and file into different fields of a database. The struggle is tweaking the parameters for parsing, ensuring i get all the info correctly.

the format is shown below:

WHITESPACE HERE of unknown length.
K       PA   DETAILS 
2 4565434   i need this sentace as 开发者_如何学运维one DB record
2 4456788   and this one 
5 4879870   as well as this one, content will vary! 

X Max - there sometimes is a line beginning with 'Max' here which i don't need
There is a Line here that i do not need!
WHITESPACE HERE of unknown length.

The tough parts were 1) Getting rid of whitespace, and 2)defining the fields from each other, see my best attempt, below:

dict = {}
    XX = (open("XX.txt", "r")).readlines()

    for line in XX:
            if line.isspace():
            pass
        elif line.startswith('There is'):
            pass
        elif line.startswith('Max', 2):
            pass
        elif line.startswith('K'):
            pass
        else:
            for word in line.split():
                if word.startswith('4'):                    
                    tmp_PA = word
                elif word == "1" or word == "2" or word == "3" or word == "4" or word == "5":
                    tmp_K = word
                else:
                    tmp_DETAILS = word
            cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',(tmp_PA,tmp_K,tmp_DETAILS))

At the minute, i can pull the K & PA fields no problem using this, however my DETAILS is only pulling one word, i need the entire sentance, or at least 25 chars of it.

Thanks very much for reading and I hope you can help! :)

K

1. You are splitting the whole line into words. You need to split into first word, second word and the rest. Like line.split(None, 2).

2. It would probably use regular expressions. And use the oposite logic, that is if it starts with number 1 through 5, use it, otherwise pass. Like:

   pattern = re.compile(r'([12345])\s+\(d+)\s+\(.*\S)')
   f = open('XX.txt', 'r') # No calling readlines; lazy iteration is better
   for line in f:
       m = pattern.match(line)
       if m:
           cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
               (m.group(2), m.group(1), m.group(3)))
   

Oh, and of course, you should be using prepared statement. Parsing SQL is orders of magnitude slower than executing it.

If I understand correctly your file format, you can try this script

filename = 'bug.txt'
f = file(filename,'r')

foundHeaders = False

records = []

for rawline in f:
    line = rawline.strip()

    if not foundHeaders:
        tokens = line.split()
        if tokens == ['K','PA','DETAILS']:
            foundHeaders = True
        continue

    else:
        tokens = line.split(None,2)
        if len(tokens) != 3:
            break

        try:
            K = int(tokens[0])
            PA = int(tokens[1])
        except ValueError:
            break

        records.append((K,PA,tokens[2]))


f.close()

for r in records:
    print r # replace this by your DB insertion code

This will start reading the records when it encounters the header line, and stop as soon as the format of the line is no longer (K,PA,description).

Hope this helps.

Here is my attempt using re

import re

stuff = open("source", "r").readlines()

whitey = re.compile(r"^[\s]+$")
header = re.compile(r"K       PA   DETAILS")
juicy_info = re.compile(r"^(?P<first>[\d])\s(?P<second>[\d]+)\s(?P<third>.+)$")

for line in stuff:
    if whitey.match(line):
        pass
    elif header.match(line):
        pass
    elif juicy_info.match(line):
        result = juicy_info.search(line)
        print result.group('third')
        print result.group('second')
        print result.group('first')

Using re I can pull the data out and manipulate it on a whim. If you only need the juicy info lines, you can actually take out all the other checks, making this a REALLY concise script.

import re

stuff = open("source", "r").readlines()

#create a regular expression using subpatterns. 
#'first, 'second' and 'third' are our own tags ,
# we could call them Adam, Betty, etc.
juicy_info = re.compile(r"^(?P<first>[\d])\s(?P<second>[\d]+)\s(?P<third>.+)$")

for line in stuff:
    result = juicy_info.search(line)
    if result:#do stuff with data here just use the tag we declared earlier.            
        print result.group('third')
        print result.group('second')
        print result.group('first')

import re


reg = re.compile('K[ \t]+PA[ \t]+DETAILS[ \t]*\r?\n'\
                 + 3*'([1-5])[ \t]+(\d+)[ \t]*([^\r\n]+?)[ \t]*\r?\n')

with open('XX.txt') as f:

    mat = reg.search(f.read())

    for tripl in ((2,1,3),(5,4,6),(8,7,9)):

        cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
                   mat.group(*tripl)

I prefer to use [ \t] instead of \s because \s matches the following characters:
blank , '\f', '\n', '\r', '\t', '\v'
and I don't see any reason to use a symbol representing more that what is to be matched, with risks to match erratic newlines at places where they shouldn't be

Edit

It may be sufficient to do:

import re

reg = re.compile(r'^([1-5])[ \t]+(\d+)[ \t]*([^\r\n]+?)[ \t]*$',re.MULTILINE)

with open('XX.txt') as f:
    for mat in reg.finditer(f.read()):
        cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
                   mat.group(2,1,3)