Efficient merging of multiple, large xml files into one

I searched the web and I searched stackoverflow up and down. No solution. Although I found solutions how to do this within pure xslt here.

But the problem is that the resulting xml will be several hundred MB large. So I must do this with SAX in Java. (please no xslt solution, although I tagged it with xslt ;-))

Let me explain with more detail. I have several multiple xml files (preferable InputSteam's) which should be parsed. The files or InputStream's looks like

inputstream1

<root>
  <doc>
    <tag>test1</tag>
  </doc>
  <doc>
    <tag>test2</tag>
  </doc>
  ...开发者_开发百科
</root>

inputstream2

<root>
  <doc>
    <tag>test3</tag>
  </doc>
  <doc>
    <tag>test4</tag>
  </doc>
  ...
</root>

inputstream1+inputstream2+...+inputstreamN = resulting xml. It will look like

<root>
  <doc>
    <tag>test1</tag>
  </doc>
  <doc>
    <tag>test2</tag>
  </doc>
  ...
   <doc>
    <tag>test3</tag>
  </doc>
  <doc>
    <tag>test4</tag>
  </doc>
  ...
</root>

Do someone has a solution or a link for this? Is this possible via implementing a custom InputSource or should I use a custom ContentHandler? Or is this possible with joost/stx?

The nice thing if I could use a ContentHandler would be that I could apply some minor transformations (I already implemented this). But then the problem is that I don't know a way to pass multiple files or InputStream's as InputSource:

XMLReader xmlReader = XMLReaderFactory.createXMLReader();
xmlReader.setContentHandler(customHandler);
xmlReader.parse(getInputSource()); // only one InputStream possible

or should I parse the InputStreams directly within my ContentHandler?

I haven't done this myself, but I recalled seeing an IBM developerworks article that looked like it made this pretty easy.

It's a bit old now, but try http://www.ibm.com/developerworks/xml/library/x-tipstx5/index.html

This is StAX instead of SAX. I'm not sure current JDKs include StAX. If not you can probably get it from http://stax.codehaus.org/

You may want to have a look at the pay-for version of Saxon. It can handle on-the-fly XSLT not needing the full DOM in memory.

I finally managed this via the following snippet:

  finalHandler = new StreamResult(new OutputStreamWriter(System.out));
  // customHandler extends DefaultHandler
  CustomTransformerHandler customHandler = new CustomTransformerHandler(
         finalHandler);
  customHandler.startDocumentExplicitly();
  InputStream is = null;
  while ((is = customHandler.createNextInputStream()) != null) {
    // multiple inputStream parsing
    XMLReader myReader = XMLReaderFactory.createXMLReader();
    myReader.setContentHandler(customHandler);
    myReader.parse(new InputSource(is));
  }
  customHandler.endDocumentExplicitly();

The important part was to leave the startDocument and endDocument methods empty. All other methods (characters, startElement, endElement) will be redirected to the finalHandler. The customHandler.createNextInputStream method returns null if all inputstreams are read.

the most effective way to merge files are to use byte level cut and paste feature offered by VTD-XML, AFAIK. You take both files, parse them into VTDNav objects, then instantiate an XMLModifier object, grab the fragments from the second file, and insert them into the first file... that got to be far more efficient than SAX.. Also the resultant XML gets written direction onto a file -- there is no need to store it in memory. Below is the complete code in less than 20 lines...

import com.ximpleware.*;
import java.io.*;

public class merge {
    // merge second.xml into first.xml assuming the same encoding
    public static void main(String[] s) throws VTDException, IOException{
        VTDGen vg = new VTDGen();
        if (!vg.parseFile("d:\\xml\\first.xml", false))
            return;
        VTDNav vn1=vg.getNav();
        if(!vg.parseFile("d:\\xml\\second.xml", false))
            return;
        VTDNav vn2 = vg.getNav();
        XMLModifier xm = new XMLModifier(vn1);
        long l = vn2.getContentFragment();
        xm.insertBeforeTail(vn2, l);
        xm.output("d:\\xml\\merged.xml");   
    }
}