php memory limit 2000M exhausted

Am trying to upload Do not call list xml file to MySql DB. The file size is 256MB. It gets uploaded but when its starts processing am looping the phone numbers and concatenating 1000 phone numbers in a variable. When it reaches the limit of 1000, am creating a INSERT query and executing the query. BUt the problem is after inserting 3,50,000 records it say "Fatal error: Allowed memory size of 2097152000 bytes exhausted (tried to allocate 24 bytes) in /var/domains/htdocs/aim/leads/dnc on line 1308)".

Below is the code snippet.

                    if(($fp = fopen($destination,"r"))) {

                        $buffer = fgets($fp);
                        $objXml = new XML($buffer);
                        $lists = Set::reverse($objXml);

                        $buffer = fgets($fp);
                        $objXml = new XML($buffer);
                        $acc = Set::reverse($objXml);

                        try {
                            while(!feof($fp)){
                                $counter = 0;
                                $phonesNo = "";

                                while(!feof($fp) && ($buffer = fgets($fp)) && $counter != 1000){

                                    $objXml = "";
                                    $arrXml = "";

                                    $objXml = new XML($buffer);

 开发者_如何学编程                                   $arrXml = Set::reverse($objXml);


                                    $counter++;
                                    if(isset($arrXml['Ph']['val']))
                                        $phonesNo .= "(".$acc['Ac']['val'].$arrXml['Ph']['val']."),";

                                }

                                $phonesNo = substr($phonesNo,0,-1);
                                $sql = "INSERT INTO do_not_call_lists (`phone`) VALUES " .$phonesNo;

                                unset($phonesNo);
                                $this->Lead->query($sql);
                            }
                        }catch (Exception $ex){
                            trigger_error("ERROR: ". $ex->getMessage(). "  Trace: ".$ex->getTrace());
                        }

                        $this->Session->setFlash('File imported successfully.','default',array('class'=>'sucessMsg'));

                     }

Thanks a lot in advance.

I think what he means is not how to extend the limits, but why the hell is this allocating 2GB of RAM?

From the looks of it, I'm assuming you're using simplexml to load the data. SimpleXML is a memory overkill and it can't be used for large files without exactly this happening. You should instead try using:

1. event based SAX parser (less memory usage than the 2nd one, but much harder to operate)
2. pull based XMLReader

FYI, I've been using SAX to load about 200MB XML file with peak memory usage about 5MB, so yes, the memory consumption is uncomparable.

its due to memory limit by default set in ur ini, You can increase it via .htaccess or via ini_Set. some additional parameters are also required as your code requires upload part too. i have given you the method to resolve it via .htaccess. you can also use it as ini_set by just replacing php_value with ini_set in your php page. hope it helps you

php_value post_max_size 1000M
php_value upload_max_filesize 2500M
php_value max_execution_time 6000000
php_value max_input_time 6000000
php_value memory_limit 2500M

Parsing XML with loads of data is quite a heavy task, so maybe it will be better to use regular expressions here. Try modifying your loop like this:

try {
    while(!feof($fp)){
        $counter = 0;
        $buffer = '';
        while(!feof($fp) && ($buffer .= fgets($fp)) && $counter++ < 1000);
        // read 1000 rows into buffer
        $matches = array();
        if (preg_match_all('/<Ph[^>]*>(\d+)</Ph>/i', $buffer, $matches) > 0) {
            $sql = 'INSERT INTO do_not_call_lists (`phone`) VALUES (' . implode('), (', $matches[1]) . ')';
            $this->Lead->query($sql);
        }
    }
}catch (Exception $ex){
    trigger_error("ERROR: ". $ex->getMessage(). "  Trace: ".$ex->getTrace());
}