I'm planning to upload a billion records taken from ~750 files (each ~250MB) to a db using django's ORM. Currently each file takes ~20min to process, and I was wondering if there's any way to accelerate this process.
I've taken the following measures:
What else can I do to speed things up? Here are some of my thoughts:
Any pointers regarding these items or any other idea would be welcome :)
Django 1.4 provides a
bulk_create() method on the QuerySet object, see: