Doctrine2 ORM - persist operations taking a long time

Question

I have a Symfony2 application that uses Doctrine2 ORM.

I am trying to create entities from an XML file, and then persist the entities. A typical XML file might contain a few thousand records that need to be persisted. Each record in the XML does not map directly onto a single entity, but onto an entity and then some other entities that are on the "many" side of a one-to-many relationship.

I can create the entity from the XML elements, but it comes to running "persist" on the entity, each operation takes around 2 seconds on my machine. With several thousand records being imported from the XML file, this is too slow for our needs.

Could anyone offer any help?

Answer 1

See Batch Processing in Doctrine documentation. The idea is to call persist() on each new entity but use flush() only after a group of n entities are persisted. It will take less time than calling persist() then flush() for every entity.

For example:

$batchSize = 20;
for ($i = 1; $i <= 10000; ++$i) {
    $user = new CmsUser;
    $user->setStatus('user');
    $user->setUsername('user' . $i);
    $user->setName('Mr.Smith-' . $i);
    $em->persist($user);
    if (($i % $batchSize) === 0) {
        $em->flush();
    }
}
$em->flush();

I removed clear() since it will detach all the entities. It's a problem if for example you used a foreach() {} with entities, since Doctrine2 would detach the entities, the loop would be broken.

Without using clear(), Doctrine2 keep in memory all the persisted entities, it can cause an error if it takes more memory than PHP can use.

If you are iterating the loop on something else than Doctrine repositories, then you can call clear() after flush().