开发者

How to parse xml data efficiently?

I have 2 questions:

1-I need to parse XML file and insert the data into mysql database. Let's say that the file is around 250 kB (but it could be even larger) and it has a lot of subnodes, so I need at least 3 tables. I parsed the xml with SimpleXml and successfully inserted all the data into db. But for this exact file it took around 160s which seems to me a lot. Is there a method to do better, in lesser time?

And another question is that I need to get the XML file from an URL and save to the server, and I'm not sure how to do this ...

Thanks for your answers.

The code for parsing xml

function parse_xml($file=""){
  global $database;
  if(file_exists($file) && !empty($file)){
      $sport = new SimpleXMLElement($file, null, true);    
      $count = count($sport->OddsObject)-1;
      $listAttr = array();
      $start_time = time();
      for($i=0; $i <= $count; $i++){
          $countMatch = count($sport->OddsObject[$i]->Matches->Match)-1;
          //echo $countMatch; 
          for($k=0; $k <= $countMatch; $k++){           
              $OOdata = $sport->OddsObject[$i]->children();
              $columns = array();
              $data = array();
              foreach($OOdata as $key => $value){            
                  if($key != "Matches"){
                      //$listAttr[$i][$key] = $attr;
                      $columns[] = $key;
                      if ($value != "") {
                          $data[] = "'" . $database->escape_value($value) . "'";
                    } else {
                         $data[] = "NULL";
                    }
                }
            }        

            //get matches: MatchId, Date, HomeTeam, AwayTeam
            $Mdata = $sport->OddsObject[$i]->Matches->Match[$k]->children();     
            foreach ( $Mda开发者_JAVA技巧ta as $key => $value) {
                if($key != "OddsData"){    
                    $columns[] = $key;
                    if ($value != "") {
                      $data[] = "'" . $database->escape_value($value) . "'";
                    } else {
                      $data[] = "NULL";
                    }    
                }
            }                      
            $cols = strtolower(implode(",",$columns));
            $values = implode(",",$data);
            $sql = "INSERT INTO sports($cols) values(".$values.")";
            if($database->query($sql)) {
                $last_id = $database->insert_id();

                $countData = count($sport->OddsObject[$i]->Matches->Match[$k]->OddsData)-1;
                for($t=0; $t <= $countData; $t++){
                    //get OddsData: Home-,Draw-, -Away ...
                    $ODdata = $sport->OddsObject[$i]->Matches->Match[$k]->OddsData[$t]->children();
                    foreach($ODdata as $key=>$attr){
                        $MID = $last_id;
                        $new_bet = Bet::make($attr->getName(),$attr, $MID);
                        $new_bet->save(); 

                    }                    
                }
            }
        }
        $end_time = time() - $start_time;
    }    
    return $end_time;
}
else{
    die("The file doesn't exist.");
}
}


A pretty easy way to get a file from a url and write it is file_get_contents() and file_put_contents().

SimpleXML should be pretty efficient and fast on a file thats only 250kb. Your slowness might be with your database inserts. Try grouping your inserts to the database. I've found that running 50 inserts at a time usually works the best(this depends on the row size though). That will probably speed up the whole process quite a bit.


I assume you're parsing it with

$dom = new DOMDocument();   
... 
// read and insert into db

DOM can use a significant amount of memory and cpu compared to SAX parser, you might try commenting out database code and running it to see if it uses too much CPU and RAM, if so you may want to re-code it with SAX parser, as shown here.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜