Parsing large XML files?

I have 2 xml files 1 with 115mb size and another with 34mb size.

Wiile reading file A there is 1 field called desc that relations it with file B where I retrieve the field id from file B where desc.file A is iqual to name.file B.

file A is already too big then I have to search inside file B and it takes a very long time to complete.

How could I speed up this proccess or what would be a better approch to do it ?

current code I am using:

#!/usr/bin/perl

use strict;
use warnings;

use XML::Simple qw(:strict XMLin);

my $npcs = XMLin('Client/client_npcs.xml', KeyAttr => { }, ForceArray => [ 'npc_client' ]);
my $strings = XMLin('Client/client_strings.xml', KeyAttr => { }, ForceArray => [ 'string' ]);

my ($nameid,$rank);

open (my $fh, '>>', 'Output/npc_templates.xml');
print $fh "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<npc_templates xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:noNamespaceSchemaLocation=\"npcs.xsd\">\n";
foreach my $npc ( @{ $npcs->{npc_client} } ) {
        if (defined $npc->{desc}) {
                foreach my $string (@{$strings->{string}}) {
                        if (defined $string->{name} && $string->{name} =~ /$npc->{desc}/i) {
                                $nameid = $string->{id};
                                last;
                        }
                }
        } else {
                $nameid = "";
        }

        if (defined $npc->{hpgauge_level} && $npc->{hpgauge_level} > 25 && $npc->{hpgauge_level} < 28) {
            $rank = 'LEGENDARY';
        } elsif (defined $npc->{hpgauge_level} && $npc->{hpgauge_level} > 21 && $npc->{hpgauge_level} < 23) {
            $rank = 'HERO';
        } elsif (defined $npc->{hpgauge_level} && $npc->{hpgauge_level} > 10 && $npc->{hpgauge_level} < 15) {
            $rank = 'ELITE';
        } elsif (defined $npc->{hpgauge_level} && $npc->{hpgauge_level} > 0 && $npc->{hpgauge_level} < 11) {
            $rank = 'NORMAL';
        } else {
            $rank = $gauge;
        }

        print $fh qq|\t<npc_template npc_id="$npc->{id}" name="$npc->{name}" name_id="$nameid" height="$npc->{scale}" rank="$rank" tribe="$npc->{tribe}" race="$npc->{race_type}" hp_gauge="$npc->{hpgauge_level}"/>\n|;
}
print $fh "</<npc_templates>";
close($fh);

example of file A.xml:

<?xml version="1.0" encoding="utf-16"?>
<npc_clients>
  <npc_client>
    <id>200000</id>
    <name>SkillZone</name>
    <desc>STR_NPC_NO_NAME</desc>
    <dir>Monster/Worm</dir>
    <mesh>Worm</mesh>
    <material>mat_mob_reptile</material>
    <show_dmg_decal>0</show_dmg_decal>
    <ui_type>general</ui_type>
    <cursor_type>none</cursor_type>
    <hide_path>0</hide_path>
    <erect>1</erect>
    <bound_radius>
      <front>1.200000</front>
      <side>3.456000</side>
      <upper>3.000000</upper>
    </bound_radius>
    <scale>10</scale>
    <weapon_scale>100</weapon_scale>
    <altitude>0.000000</altitude>
    <stare_angle>75.000000</stare_angle>
    <stare_distance>20.000000</stare_distance>
    <move_speed_normal_walk>0.000000</move_speed_normal_walk>
    <art_org_move_speed_normal_walk>0.000000</art_org_move_speed_normal_walk>
    <move_speed_normal_run>0.000000</move_speed_normal_run>
    <move_speed_combat_run>0.000000</move_speed_combat_run>
    <art_org_speed_combat_run>0.000000</art_org_speed_comb开发者_运维问答at_run>
    <in_time>0.100000</in_time>
    <out_time>0.500000</out_time>
    <neck_angle>90.000000</neck_angle>
    <spine_angle>10.000000</spine_angle>
    <ammo_bone>Bip01 Head</ammo_bone>
    <ammo_fx>skill_stoneshard.stoneshard.ammo</ammo_fx>
    <ammo_speed>50</ammo_speed>
    <pushed_range>0.000000</pushed_range>
    <hpgauge_level>3</hpgauge_level>
    <magical_skill_boost>0</magical_skill_boost>
    <attack_delay>2000</attack_delay>
    <ai_name>SummonSkillArea</ai_name>
    <tribe>General</tribe>
    <pet_ai_name>Pet</pet_ai_name>
    <sensory_range>15.000000</sensory_range>
  </npc_client>
</npc_clients>

example of file B.xml:

<?xml version="1.0" encoding="utf-16"?>
<strings>
  <string>
    <id>350000</id>
    <name>STR_NPC_NO_NAME</name>
    <body> </body>
  </string>
</strings>

Here is example of XML::Twig usage. The main advantage is that it is not holding whole file in memory, so processing is much faster. The code below is trying to emulate operation of script from question.

use XML::Twig;

my %strings = ();
XML::Twig->new(
    twig_handlers => {
        'strings/string' => sub {
            $strings{ lc $_->first_child('name')->text }
                = $_->first_child('id')->text
        },
    }
)->parsefile('B.xml');

print "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<npc_templates xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:noNamespaceSchemaLocation=\"npcs.xsd\">\n";
XML::Twig->new(
    twig_handlers => {
        'npc_client' => sub {
            my $nameid = eval { $strings{ lc $_->first_child('desc')->text } };

            # calculate rank as needed
            my $hpgauge_level = eval { $_->first_child('hpgauge_level')->text };
            $rank = $hpgauge_level >= 28 ? 'ERROR'
                  : $hpgauge_level  > 25 ? 'LEGENDARY'
                  : $hpgauge_level  > 21 ? 'HERO'
                  : $hpgauge_level  > 10 ? 'ELITE'
                  : $hpgauge_level  >  0 ? 'NORMAL'
                  :                        $hpgauge_level;

            my $npc_id    = eval { $_->first_child('id')->text };
            my $name      = eval { $_->first_child('name')->text };
            my $tribe     = eval { $_->first_child('tribe')->text };
            my $scale     = eval { $_->first_child('scale')->text };
            my $race_type = eval { $_->first_child('race_type')->text };
            print
                qq|\t<npc_template npc_id="$npc_id" name="$name" name_id="$nameid" height="$scale" rank="$rank" tribe="$tribe" race="$race_type" hp_gauge="$hpgauge_level"/>\n|;
            $_->purge;
        }
    }
)->parsefile('A.xml');
print "</<npc_templates>";

1. Grab all the interesting 'desc' fields from file A and put them in a hash. You only have to parse it once, but if it still takes too long have a look at XML::Twig.
2. Parse file B. once and extract the stuff you need. Use the hash.

Looks like you only need parts of the xml files. XML::Twig can parse only the elements you are interested in and throw away the rest using the "twig_roots" parameter. XML::Simple is easier to get started with though..

Although I can't help you with the specifics of your Perl code, there are some general guidelines when dealing with large volumes of XML data. There are, broadly speaking, 2 kinds of XML APIs - DOM based and Stream based. Dom based API's (like XML DOM) will parse an entire XML document in to memory before the user-level API becomes "available", whereas with a stream based API (like SAX) the implementation doesn't need to parse the whole XML document. One benefit of Stream based parsers are that they typically use much less memory, as they don't need to hold the entire XML document in memory at once - this is obviously a good thing when dealing with large XML documents. Looking at the XML::Simple docs here, it's seems there may be SAX support available - have you tried this?

I'm not a perl guy, so take this with a grain of salt, but I see 2 problems:

1. The fact that you are iterating over all of the values in file B until you find the correct value for each element in file A is inefficient. Instead, you should be using some sort of map/dictionary for the values in file B.

2. It looks like you are parsing the both files in memory before you even start processing. File A would be better processed as a stream instead of loading the entire document into memory.