Search code examples
cocoacore-datacore-data-migration

Core Data migration: exceptions after changing a relationship from one entity to its parent entity

I have a Core Data model that includes a Car entity and let's say an EngineData entity. This is a one-to-one relationship.

In a new version of my app I want to add trucks. So I've created a new version of my Core Data model. I now have a Vehicle entity that is the parent entity of Car. I've added a new Truck entity that also has Vehicle as its parent entity. For EngineData, the inverse relationship was generically named object, so the destination entity just changes from Car to Vehicle.

I wasn't entirely sure this would work with lightweight migration, but I first changed it a couple of weeks ago, and until now it seemed fine. I have code that fetches and updates existing data from EngineData using a Car's engineData property, and I haven't seen any issues there. However, there's a single Core Data fetch in my app that causes a crash every time. All I have to do to cause the crash is a simple fetch of all my EngineData objects:

do {
    let request: NSFetchRequest<EngineData> = EngineData.fetchRequest()
    let objects = try context.fetch(request)
} catch {
    NSLog("Error fetching data: \(error)")
}

On the context.fetch line, I get an exception:

[error] error: Background Core Data task threw an exception.  Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
CoreData: error: Background Core Data task threw an exception.  Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)

And if I try to actually do anything with those objects, I get some more exceptions until the app crashes:

[General] An uncaught exception was raised
[General] *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10]

0   CoreFoundation                      0x00007fff8861937b __exceptionPreprocess + 171
1   libobjc.A.dylib                     0x00007fff9d40d48d objc_exception_throw + 48
2   CoreFoundation                      0x00007fff88532b5c -[__NSArrayM objectAtIndex:] + 204
3   CoreData                            0x00007fff881978ed -[NSSQLRow newObjectIDForToOne:] + 253
4   CoreData                            0x00007fff8819770f -[NSSQLRow _validateToOnes] + 399
5   CoreData                            0x00007fff88197571 -[NSSQLRow knownKeyValuesPointer] + 33
6   CoreData                            0x00007fff88191868 _prepareResultsFromResultSet + 4312
7   CoreData                            0x00007fff8818e47b newFetchedRowsForFetchPlan_MT + 3387
8   CoreData                            0x00007fff8835f6d7 _executeFetchRequest + 55
9   CoreData                            0x00007fff8828bb35 -[NSSQLFetchRequestContext executeRequestUsingConnection:] + 53
10  CoreData                            0x00007fff8832c9c8 __52-[NSSQLDefaultConnectionManager handleStoreRequest:]_block_invoke + 216
11  libdispatch.dylib                   0x000000010105478c _dispatch_client_callout + 8
12  libdispatch.dylib                   0x00000001010555ad _dispatch_barrier_sync_f_invoke + 307
13  CoreData                            0x00007fff8832c89d -[NSSQLDefaultConnectionManager handleStoreRequest:] + 237
14  CoreData                            0x00007fff88286c86 -[NSSQLCoreDispatchManager routeStoreRequest:] + 310
15  CoreData                            0x00007fff88261189 -[NSSQLCore dispatchRequest:withRetries:] + 233
16  CoreData                            0x00007fff8825e21d -[NSSQLCore processFetchRequest:inContext:] + 93
17  CoreData                            0x00007fff8817d218 -[NSSQLCore executeRequest:withContext:error:] + 568
18  CoreData                            0x00007fff882436de __65-[NSPersistentStoreCoordinator executeRequest:withContext:error:]_block_invoke + 5486
19  CoreData                            0x00007fff8823a347 -[NSPersistentStoreCoordinator _routeHeavyweightBlock:] + 407
20  CoreData                            0x00007fff8817cd9e -[NSPersistentStoreCoordinator executeRequest:withContext:error:] + 654
21  CoreData                            0x00007fff8817af51 -[NSManagedObjectContext executeFetchRequest:error:] + 593
22  libswiftCoreData.dylib              0x0000000100c91648 _TFE8CoreDataCSo22NSManagedObjectContext5fetchuRxSo20NSFetchRequestResultrfzGCSo14NSFetchRequestx_GSax_ + 56

I thought the flag -com.apple.CoreData.SQLDebug 1 might give me some useful information, but I don't see anything very helpful here:

CoreData: annotation: Connecting to sqlite database file at "/Users/name/Library/Group Containers/Folder/Database.sqlite"
CoreData: sql: pragma recursive_triggers=1
CoreData: sql: pragma journal_mode=wal
CoreData: sql: SELECT Z_VERSION, Z_UUID, Z_PLIST FROM Z_METADATA
CoreData: sql: SELECT TBL_NAME FROM SQLITE_MASTER WHERE TBL_NAME = 'Z_MODELCACHE'
CoreData: sql: SELECT 0, t0.Z_PK, t0.Z_OPT, t0.ZDATA, t0.ZOBJECT, t0.Z9_OBJECT FROM ZENGINEDATA t0 
2017-04-28 16:18:41.693548-0400 AppName[95979:11442888] [error] error: Background Core Data task threw an exception.  Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
CoreData: error: Background Core Data task threw an exception.  Exception = *** -[__NSArrayM objectAtIndex:]: index 18446744073709551615 beyond bounds [0 .. 10] and userInfo = (null)
CoreData: annotation: sql connection fetch time: 0.0065s
CoreData: annotation: fetch using NSSQLiteStatement <0x6080004805a0> on entity 'ENGINEDATA' with sql text 'SELECT 0, t0.Z_PK, t0.Z_OPT, t0.ZDATA, t0.ZOBJECT, t0.Z9_OBJECT FROM ZENGINEDATA t0 ' returned 1185 rows with values: (
    "<JUNENGINEDATA: 0x608000480eb0> (entity: ENGINEDATA; id: 0x40000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1> ; data: <fault>)",
    "<JUNENGINEDATA: 0x608000480f00> (entity: ENGINEDATA; id: 0x80000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p2> ; data: <fault>)",
    "<JUNENGINEDATA: 0x608000480f50> (entity: ENGINEDATA; id: 0xc0000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p3> ; data: <fault>)",
    ...
    "<JUNENGINEDATA: 0x600000288110> (entity: ENGINEDATA; id: 0x128c0000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1187> ; data: <fault>)",
    "<JUNENGINEDATA: 0x600000288160> (entity: ENGINEDATA; id: 0x12900000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1188> ; data: <fault>)",
    "<JUNENGINEDATA: 0x6000002881b0> (entity: ENGINEDATA; id: 0x12940000b <x-coredata://10C884E2-18EF-4DA2-BC5D-4CBD0CE7D1A6/ENGINEDATA/p1189> ; data: <fault>)"
)
CoreData: annotation: total fetch execution time: 0.1053s for 1185 rows.
2017-04-28 16:18:41.793201-0400 AppName[95979:11442649] Got results: 1185

The good news is everything in EngineData can be recreated, and I've found that if I do a batch delete of all EngineData objects, it no longer crashes (even after creating some new objects). If possible I'd greatly prefer to understand the cause of the problem though, and find a less drastic solution.

Here are some other things I've discovered:

  • I tried copying a database from another computer, and I was no able to duplicate the problem using that data. That got me thinking maybe the data was corrupt to begin with…
  • But if I go back to an old version of my app and its data, the same fetch works fine. If I do nothing other than upgrade the data model, I get the same exceptions.
  • I've tried setting a version hash modifier for the relationship, hoping that might force Core Data to clean things up when it migrates, but no luck.
  • If I set a batch size of 10, it's able to loop through 900 results (out of 1185) before the exception.
  • Using that process, I can delete good records one at a time to narrow down which ones are problematic. (I'm saving after each delete, and I kept a backup of the original database if I need to go back for further testing.)
Solution

  • After a lot of experimenting I was able to narrow it down: I have a handful of EngineData objects that reference a Car object that no longer exists. It looks like since that relationship is broken, the records aren't migrated correctly to the new data model version. If I open the .sqlite file in the Mac app Base, I can see a new Z9_OBJECT field that's set to 10 for all the good records, and NULL for the damaged ones.

    Fixing the problem was fairly easy once I discovered the cause. Using NSBatchDeleteRequest I was able to find and delete the damaged objects:

    do {
    let fetchRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "EngineData")
    fetchRequest.predicate = NSPredicate(format: "object.uuid == nil")
    let batchRequest = NSBatchDeleteRequest(fetchRequest: fetchRequest)
    try context.execute(batchRequest)
} catch {
    NSLog("Error deleting objects: \(error)")
}

    In my predicate, object is the parent object (previously a Car, now a Vehicle), and uuid is a non-optional attribute on that object. As long as the attribution isn't optional—meaning it will have a value on any undamaged object—this should successfully delete only the objects that are damaged.

    You might be expecting this to cause an exception, since it's still fetching the damaged objects! Fortunately NSBatchDeleteRequest works by running SQL statements directly on the store, without loading anything into memory—so it avoids the problem.