Importing large CSV files

Question

I'm looking at importing a CSV file, but this file is rather large.

What I'm looking to do, is 2 things:

1) Scan the CSV to validate values in particular fields
2) Once the file is valid, import

The CSV data can ONLY be inserted if the file is valid (All or nothing)

The problem is, I'm looping through twice, 1st to check the CSV fields are valid and then another for loop to save.

The problem is memory. I run out of memory (file is 100,000 rows with 45 fields)

Is there an easier way to do this and reduce memory? I'm using an AR implementation, would using PDO be quicker?

Thanks

EDIT:

       $data = array();
        // open the file and loop through
        if( ($handle = fopen('details.csv', "r")) !== FALSE) {
            $rowCounter = 0;
            while (($rowData = fgetcsv($handle, 0, ",")) !== FALSE) {
                if( 0 === $rowCounter) {
                    $headerRecord = $rowData;
                } else {
                    foreach( $rowData as $key => $value) {
                        $data[ $rowCounter - 1][$headerRecord[ $key] ] = $value;
                    }
                }
                $rowCounter++;
            }
            fclose($handle);
        }

        $errors = array();
        // loop to check through the fields for validation
        for($i=0;$i<count($data);$i++) {
            $row++;

            if(!valid_email($data[$i]['EMAIL']))) {
               $errors[] = 'Invalid Email Address';
               break;
            }

        }

        if(empty($errors)) {
          for($j=0;$j<count($assocData);$j++) {
             $row++;

             $details = new Details();

             // set the fields here
             $details->email = $data[$j]['EMAIL'];

             $details->save();
             unset($details); 
          } 
        }

Answer 1

You're already looping through the data in your first foreach. Why don't you validate the fields inside that loop, if validation passes add to an array to save and only save (in a single transaction) if the loop completes.