djangooptimizationormbulkinsert

Accelerate bulk insert using Django's ORM?


I'm planning to upload a billion records taken from ~750 files (each ~250MB) to a db using django's ORM. Currently each file takes ~20min to process, and I was wondering if there's any way to accelerate this process.

I've taken the following measures:

What else can I do to speed things up? Here are some of my thoughts:

Any pointers regarding these items or any other idea would be welcome :)


Solution

  • Django 1.4 provides a bulk_create() method on the QuerySet object, see: