Getting No space left on device (28) for resource intensive php scripts

Question

I have two PHP scripts, the parent script has a loop of 1000000 and in each loop it calls another child script with the help of shell_exec(). The child script performs 10000 insertions in mysql table in each call, echoes a response and exits after operation. The response is then used by the parent script and it performs furthur operation.

The scripts are coded to run without memory limit and also no limit in execution time.

The scripts run successfully for most part but after some time it shows this error "No space left on device (28)". On monitoring at the time of processing in phpmyadmin, I have found that out of 16Gbs of ram, 7Gbs were used constantly by the program, around 6.7Gbs were cached memory and had around 160mb of free memory. The cpu processing was also stable around 20% and connections/processes all seemed stable. Estimated mysql table was around 15Gb and I had 40Gb of free disk space before starting the operation.

Here is a sample code for reference in the parent script:

<?php
    //parent script
    error_reporting(E_ALL);
    ini_set('max_execution_time', 0);
    ini_set('memory_limit', -1);
    ini_set('max_input_vars', 10000);
    include 'db-connection-script.php';

    for($i =0; $i<1000000; $i++){
        //restart script starting
        $mySqlTableNameEscaped=escapeshellarg($mySqlTableName);

        $cmd = "/usr/bin/php child-script.php $mySqlTableNameEscaped";
        $result = shell_exec($cmd);
        unset($mySqlTableNameEscaped);

        //response taken here
        $resultArray = json_decode($result, true);
        if($resultArray['status'] == "success"){
            $returnArrayFromChildScript = $resultArray['returnArrayFromChildScript'];
            //furthur operations ......
            //......
            //......
        }
        unset($resultArray);
    }
    exit;
?>

Here is a sample code for reference in the child script:

<?php
    //child script
    error_reporting(E_ALL);
    ini_set('max_execution_time', 0);
    ini_set('memory_limit', -1);
    ini_set('max_input_vars', 10000);
    include 'db-connection-script.php';

    $status = "";
    $returnArrayFromChildScript = array();
    $mySqlTableName = (isset($_SERVER['argv'][1]) AND $_SERVER['argv'][1] != "") ? $_SERVER['argv'][1] : "";

    //mysql operation for selection and insertion of 1000 rows in mySqlTableName sql table is done here.
    //******/
    //******/
    //******/
    //returnArrayFromChildScript[] array is prepared here

    $status = "success";
    echo json_encode( array( 'status' => $status,'returnArrayFromChildScript'=>$returnArrayFromChildScript) ); 
    exit;
?>

Answer 1

I got this fixed. The problem was indeed with error logs in the server. It was consuming most of the storage space, which resulted in crash. Since this was an experimental code, PHP thrown many warnings and notices relating to some variable used. I reviewed the codes, made necessary changes, changed the logrotate and now it runs fine.