Using Regular Expressions to Import a Large Amount of Data

I need to import a large amount of building codes from a text file to a SQL database. So far I have written the following code which successfully returns the code number and title. How can I match the text after a code's title to the beginning of the next code?

Test.txt:

101.1 Title. This is an example code.

101.1.2 Local Fees. The local jurisdiction may charge fees for building permit violations per Section 300.1.

import re

file=open(r'C:\Test.txt','r')
text=file.read()

codes=re.findall('(\d{3,4}.[\d.]+?){1}\s([\w\s]+[.]){1}',text)
for code in codes:
    print code[0],code[1]

This results in:

开发者_如何学JAVA101.1 Title. I would like to have code[3] print 'This is an example code.'

101.1.2 Local Fees.

Use re.split instead of re.findall. In your case:

>>> re.split('(\d{3,4}.[\d.]+?){1}\s([\w\s]+[.]){1}',text)

['', '101.1', 'Title.', ' This is an example code.\n\n', '101.1.2', 'Local Fees.', ' The local jurisdiction may charge fees for building permit violations per Section 300.1.\n']

It looks to me that your title of your record is the text between the last number (if you're incrementing on the strings chars), and the first period that is neither preceeded by an number nor followed by one.

Using this you could take in the whole line and determine the position of the period that separates the two parts of the record and split them based upon that. Then cache the two strings as a pair, or insert them to your database outright.

If your file is just building codes that you want to extract, and doesn't contain any other data that is of no use, I'd suggest scrapping the regular expressions and using this approach.

I would use the folowing (not tested)

codes=re.findall('^(\d{3,4}.[\d.]+?)\s([\w\s]+[.])\s+(.*)$',text,re.MULTILINE)

Note that {1} is useless

import sys
import re

SECTION = r'\d{3,4}\.[\d.]*'
LABEL = r'[^.\r\n]*'
TITLE = re.compile(r'^({section})\s*({label})\W*'.format(section=SECTION, label=LABEL), re.MULTILINE)

def process(fname):
    with open(fname, 'r') as inf:
        txt = inf.read()
    res = TITLE.split(txt)
    it = iter(res)
    it.next()         # discard leading text
    return zip(it, it, it)

def main():
    args = sys.argv[1:] or ['c:/test.txt']
    for fname in args:
        res = process(fname)
        # do something with res

if __name__=="__main__":
    main()

Run against your test.txt, this returns

[
    ('101.1', 'Title', 'This is an example code.\n\n'),
    ('101.1.2', 'Local Fees', 'The local jurisdiction may charge fees for building permit violations per Section 300.1.\n\n')
]