Import large amount of data into coredata with unique constraint

Question

I have an entity Word with keyword is a uniqueConstraint of type String. Now I have to import large amount (>100000 records) of Words with new keywords from a text file. I tried to create coredata NSManagedObject and insert into context with NSPolicyObjectTrump to merge new keywords with the old ones. But unfortunately, my saving project on context goes slow (2-3 minutes) because of coredata’s resolving conflict process. So which is the best approach to import such a large amount of project and make sure that no uniqueConstraint on coredata be violated? Should I pre-create sqlite and how to migrate sqlite files with the same object model? Thank you

Answer 1

1. Split the array into many slices. Each slice should be around 200 items. You should test and tweak this number to find the right batch size for your application, but this number has worked for me in my own applications.

2. Make an operation queue and an operation for each slice, so you can process each slice, one at a time

3. For each slice that needs to be processed:

   • Fetch all entities with that have the same ID of the element you are inserting (with ONE fetch request)
   • Inserted the fetched elements into a dictionary where the key is the unique Id and the value is the mananagedObject that you fetched
   • For each element see if you already have a duplicate. If you do then update or ignore it (depending on your application logic). If you don't have one then insert it
   • Save the context before moving on to the next batch