PHP: How to optimise looping through a CSV and writing a line to a file dependent on value in column

Question

I have the following code which takes a $filename and loops through it. If the 9th column is in an array of values (not shown here), I ignore it.

Otherwise I write the row to a file which name is based on the 3rd column.

if (($handle = fopen($filename, "r")) !== FALSE) {
    fgetcsv($handle);
    while (($line = fgetcsv($handle, 2000, ";")) !== FALSE) {
        if (!in_array($line[8], $exclude)) {
            $d = str_replace('/','',$line[2]);
            $f = fopen($base.$d.'.csv', "a");
            fputcsv($f, $line);
            fclose($f);
            unset($line);
        }
    }
    fclose($handle);
}

This works fine. However it's very slow. I have a 200Mb CSV it's looping through.

My question is whether it can be optimised and/or whether I am doing anything tragically wrong?

Thanks

Answer 1

Opening and closing files is always an expensive operation, so reducing this would help as you open and close a file for every row in the input file.

This code keeps an array of the files opened and each time checks if it is already open, if so just use the stored handle, if not open and store the new handle. Then at the end of the code it loops through all of the open files and closes them all...

if (($handle = fopen($filename, "r")) !== FALSE) {
    $outHandles = [];
    fgetcsv($handle);
    while (($line = fgetcsv($handle, 2000, ";")) !== FALSE) {
        if (!in_array($line[8], $exclude)) {
            $d = str_replace('/','',$line[2]);
            if ( isset($outHandles[$d]) )   {
                $f = $outHandles[$d];
            }
            else    {
                $f = fopen($base.$d.'.csv', "a");
                $outHandles[$d] = $f;
            }
            fputcsv($f, $line);
            unset($line);
        }
    }
    fclose($handle);
    foreach ( $outHandles as $file )    {
        fclose($file);
    }
}