Iterate over SAX

I have an xml like this (just an example):

<xml>
  <page>
    <lol>
    </lol>
    <lel>
    </lel>
  </page>
  &l开发者_StackOverflowt;page>
    <lol>
    </lol>
    <lel>
    </lel>
  </page>
  <page>
    <lol>
    </lol>
    <lel>
    </lel>
  </page>
</xml>

I need a way to do something like this:

#Sax code

for page in something:
  parse(page)

How i can do this with sax?

The xml file contains 30GB of data.

Do not use SAX, use ElementTree instead:

from xml.etree import cElementTree as ET

for event, elem in ET.iterparse("/path/to/your/file"):
    if elem.tag == 'page':
        # do your processing
        elem.clear()

The elem.clear() call is important, otherwise you will keep all the processed elements in memory and eventually consume all your RAM, too. The element objects are light-weight DOM-like objects, so they are quite easy to use, as compared to SAX.

If the individual page elements are too large already to fit your memory, you will have to revert to SAX, but I assume from your example that there are many small page elements rather than a few large ones.

The most efficent and pythonic way to do this with xml.sax is to use the parser.feed() method.

Example:

parser = xml.sax.make_parser()
parser.setContentHandler(YourContentHandler)

f = open('terribly_large.xml', 'r')
for line in f.xreadlines():
    parser.feed(line)

This ensures that you're both incrementally reading the file, and incrementally parsing it.

The resulting memory footprint should be minimal.

You could use the sax parser in a thread. When it detects a full fage it pushes it to a queue. In your main thread, iterate over the queue.

use Dom instead of Sax , sax keep fire event when it occur interest stuff like start element or text , but if you want to iterate over the file use dom this link may help you .

UPDATE:

with 30GB you must use SAX