Importing HUGE JSON-encoded arrays without hitting memory constraints

I have a file which contains two JSON arrays; one is holding the column names which has 4 values and another array which contains 10,000+ record values.

I am using Symfony and Propel; while using json_decode it throws an allowed memory size exhausted. In my php.ini I have specified the maximum size to 500 MB, but the file executed for 10 seconds and threw the error.

the data file contains

{
"columns_map":["Name","Age","Address","State"],
"rows_map":{"1":["value1","value2","value3","value4"],
"4":["value1","value2","value3","value4"]
"6":["value1","value2","value3","value4"].......upto 10,000 and more records
}
}

in my Symfony page i have this code

$file = "path to the file";
$content = file_get_contents($file);
$array = json_decode($content);

I want to store the file array values into a PHP array and process, and I want to read r开发者_StackOverflowegardless of the memory limit set in php.ini.

I want to store all the values at once or spit the file and store (e.g. reading the first 1000 records and looping upto the end, but how read the first 1000 records in the rows_map array?).

Make sure you're updating the correct php.ini (there are usually separate files on linux systems for Apache, CGI and CLI). You can ensure that you're allowed memory value is being updated by checking the return value of ini_get('memory_limit') in the same context. And don't forget to restart your web server if you're running Apache or some CGI server.

Even 10k items should not exhaust 500MB of memory; if it really is, you'll likely run into the same problem trying to parse it on your own. It's not practical to read and parse chunks of raw JSON strings. Pick a format that's better-suited, insert the data into a database, or write the data in chunks to separate files and parse each one separately.

Is it possible to store the files separately? Then it's already a lot easier. Regard for example the following structure:

• 1.json (first 1000 rows + columnmap)
• 2.json (second 1000 rows + columnmap )

Another problem might be propel. I've observed a similar problem with Doctrine, which compelled me to use plain PDO to insert objects. Doctrine would clog up all the memory and cpu, while prepared statements with PDO could easily handle this kind of volume.

Another option is to use CSV (it's very 1980s, I know). But it should allow you to read it per line.

I solved it by creating a new class of my own with encode and decode functionalities