How can I store large chunks into json file without memory crash?

I have an array $table and it contains 6000 items. When I want to convert this array to json and store it into a file, my memory crashes. So I had the idea to break the array into chunks of parts with 500 items:

$table = array_chunk($table, 500);

$table_json = $serializer->serialize($table[0], 'json', $context);

$myfile = fopen($_SERVER['DOCUMENT_ROOT']."/files/myfile.json", "w") or die("Unable to open file!");
file_put_contents($_SERVER['DOCUMENT_ROOT']."/files/myfile.json", $table_json);

This runs fast now, but of course now only 500 items are stored. Is there a way to add the other parts of the $table array without memory crash?

Solution

You could do something like this as you mentioned you know how to use array_chunk(); Let's look into simplifying the process with a built-in PHP function called array_chunk();

We'll be using HTML tables for design which isn't recommended. The task is better accomplished with CSS, you can follow same way without HTML and CSS.

Our table :

id  datePosted  firstName   lastName    pictureName          anotherColumn
1   2013-07-01  John        Smith       SmithJohn.jpg        anotherValue
2   2013-05-06  Elroy       Johnson     JohnsonElroy.jpg     anotherValue
3   2013-06-18  Jake        Bible       BibleJake.jpg        anotherValue
4   2013-07-17  Steve       Stevenson   StevensonSteve.jpg   anotherValue
5   2013-04-08  Bill        Smith       SmithBill2.jpg       anotherValue

Building HTML Tables

PDO query is used to grab the database information with prepared statements, Note that the loop only generates the code to display the columns. The tests to detect where the rows begin and end are unnecessary.

//INITIALIZE VARIABLES
$colsToDisplay = 3;
$htmlOutput    = array();

//GET PICTURE LIST
$sql = "SELECT datePosted, firstName, lastName, pictureName FROM pictureList ORDER BY datePosted DESC";
$stmt = $pdo->prepare($sql);
$stmt->execute();
while($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
     $htmlOutput[] = "<td><img src='images/{$row['pictureName']}' alt='' /><br />{$row['firstName']} {$row['lastName']}</td>";
}

Once the loop is done, the array containing the column information can be broken into groups of three... or whatever value was assigned to $colsToDisplay, What we did here ? we tooks 3 columns from table, So divide table in two parts.

while($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
     //...
}

//BREAK THE COLUMNS INTO GROUPS
$htmlOutput = array_chunk($htmlOutput, $colsToDisplay);

All that's left is to display the table information. Note that array_chunk() creates a multi-dimensional array. The foreach loop is used to process the groups of columns. Each group is assigned to $currRow which contains an array of columns for the current row. The implode() function is used to quickly display the columns as a string.

Lets continue :

//BREAK THE COLUMNS INTO GROUPS
$htmlOutput = array_chunk($htmlOutput, $colsToDisplay);

//DISPLAY TABLE
print '<table>';
foreach($htmlOutput as $currRow) {
     print '<tr>' . implode('', $currRow) . '</tr>';
}
print '</table>';

Checking For Missing Column Tags

One thing you may have noticed is the code to add missing columns was left out. In other words, this example results in an HTML table where the last row only has two columns.

Apparently, the missing columns aren't needed according to the W3C Markup Validation Service… so they weren't included. However, they can be added by running the following code right before array_chunk() is called.

$colsDifference = count($htmlOutput) % $colsToDisplay;
if($colsDifference) {
     while($colsDifference < $colsToDisplay) {           
        $htmlOutput[] = '<td></td>';           
        $colsDifference++;
    } 
}

Final Code :

    //INITIALIZE VARIABLES
    $colsToDisplay = 3;
    $htmlOutput    = array();


    //GET PICTURE LIST
    $sql = "SELECT datePosted, firstName, lastName, pictureName FROM pictureList ORDER BY datePosted DESC";
    $stmt = $pdo->prepare($sql);
    $stmt->execute();
    while($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
         $htmlOutput[] = "<td><img src='images/{$row['pictureName']}' alt='' /><br />{$row['firstName']} {$row['lastName']}</td>";
    }

    //OPTIONAL CODE
    //IF NEEDED, ADD MISSING COLUMNS
    $colsDifference = count($htmlOutput) % $colsToDisplay;
    if($colsDifference) {
         while($colsDifference < $colsToDisplay) {           
            $htmlOutput[] = '<td></td>';           
            $colsDifference++;      
        }
    } 
        //END: OPTIONAL CODE  
        //BREAK THE COLUMNS INTO GROUPS 
   $htmlOutput = array_chunk($htmlOutput, $colsToDisplay);   
        //DISPLAY TABLE print '<table>'; 

    foreach($htmlOutput as $currRow) {      
        print '<tr>' . implode('', $currRow) . '</tr>'; 
    } 
    print '</table>';

Idea Behind This Tutorial :

You dont need to create a file and write arrays into that file to display in datatable.
If you still wants that you can (divide table in two parts) to create two arrays and than write into file, and than use array_push(); or array_merge(); to get both arrays into one.

I might have bad explanition please forgive for mistakes, Hope this help you in your coding life :)