Python: parsing XML document while preserving entities

I wanted to ask what known existing Python 2.x libraries there are for parsing an XML document with built-in DTD without automatically expanding the entities. (File in question for those curious: JMdict.)

It seems lxml has some option for not parsing the entities, but last I tried, the entities just ended up being converted to blanks. I just googled this and found pxdom as another alternative whic开发者_JAVA百科h I may try, but since it's pure Python it seems far slower than I'd like.

Anything else out there?

lxml is mentioned in the question and it does what you want as far as I can tell. Test code:

from lxml import etree

XML = """
<!DOCTYPE root [
<!ENTITY abc "123">
]>
<root>
&abc;
</root>"""

parser = etree.XMLParser(resolve_entities=False)

root = etree.fromstring(XML, parser)
print "Entity not resolved:"
print etree.tostring(root)
print

print "Entity resolved:"
root = etree.fromstring(XML)
print etree.tostring(root)

Output:

Entity not resolved:
<root>
&abc;
</root>

Entity resolved:
<root>
123
</root>

It seems that the use case is rather abnormal; not expanding entities seems to go against the way parsers are generally supposed to work according to the XML spec.

So, I think it's easiest to just kludge this perhaps. I've manually extracted the tags via re.finditer, and have made a dictionary of the mappings. From here, it's just a matter of scanning the parsed output and doing the right thing for my app. Good enough for my use case I think.

For one, BeautifulStoneSoup from BeautifulSoup won't expand entities by default.

Won't probably be fast or efficient for your use case though, since it is geared towards a different kind of usage (handling all sorts of ill-formed and broken markup).

I had a similar problem to solve. I needed to read a TEI XML file containing entities like

&some_exotic_char;

which were declared in a separate DTD file. The task was to add some attribute in certain Tags and write the modified file while preserving the special entities and not messing up the XML layout.

BeautifulSoup worked nice until I wanted to write out the XML file again:

with open('outfile.xml','w') as outfile:
    outfile.write(soup.prettify())

It then would not leave the entities «as is», but instead expand them into utf8-characters, which was not what I wanted. Plus, it messed up the original XML layout, regardless of the prettify-method (without it, it is even worse).

Finally I gave up and finally found a good solution using Perl and XML::LibXML. With the method

$parser->expand_entities(0);

entities will not get expanded. And writing the XML back to a file will keep the original layout intact.

use XML::LibXML;
my $parser = new XML::LibXML;
$parser->validation(0);
$parser->load_ext_dtd(1);
$parser->expand_entities(0);
my $doc  = $parser->parse_file('infile.xml');

... # do whatever you need to do

open my $out, '>', 'outfile.xml';
binmode $out;
print $out $doc->toString();
close $out;

Perl's XML::LibXML saved my day.