Program runs out of memory reading a large TYPO3 Extbase repository

Question

I'm writing an extension function in TYPO3 CMS 6.2 Extbase that must process every object in a large repository. My function works fine if I have about 10,000 objects, but runs out of memory if I have over about 20,000 objects. How can I handle the larger repository?

$importRecordsCount = $this->importRecordRepository->countAll();
for ($id = 1; $id <= $importRecordsCount; $id++) {
    $importRecord = $this->importRecordRepository->findByUid($id);
    /* Do things to other models based on properties of $importRecord */
}

The program exceeds memory near ..\GeneralUtility.php:4427 in TYPO3\CMS\Core\Utility\GeneralUtility::instantiateClass( ) after passing through the findByUid() line, above. It took 117 seconds to reach this error during my latest test. The error message is:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4194304 bytes) in ... \typo3\sysext\core\Classes\Utility\GeneralUtility.php on line 4448

If it matters, I do not use @lazy because of some of the processing I do later in the function.

Answer 1

According to official TYPO3 website, it is recommended 256M memory limit instead of 128M: Source

So my first suggestion would be trying to do that first and it might solve your problem now. Also you should use importRecordRepository->findAll(); instead of fetching each record by iterating uid, since someone might have deleted some records.