Extremely Massive and Continuous Import to Core Data Efficiently

Question

Questions: How do i release the memory used by the NSManagedObjectContext(i guess) when the number of records to be inserted to Core Data are unforeseeable, such that the memory can be efficiently used?

Here is my case: I have a bluetooth device that will continuously send twelve sets of integer to the iOS Device on every 0.00125 seconds(the minimum interval, maximum case will be 0.002 seconds) , i should then store those integer into CoreData with the timestamp.

Data Objects and Association:

When the process start, a header record(NSManagedObject) is created as the key to retrieve the batch of received data from the bluetooth device. The object is retained as strong property during the whole period of data receiving and will remove from the property( probably set nil) when the process is ended.

Design of the NSManagedObjectContext:

All of the three ManagedObjectContext are a singleton object stored in AppDelegate

The code for creating the ManagedObjectContext are listed below:

- (NSManagedObjectContext *)masterManagedObjectContext {
    // Returns the managed object context for the application (which is already bound to the persistent store coordinator for the application.)
    if (_masterManagedObjectContext != nil) {
        return _masterManagedObjectContext;
    }
    NSPersistentStoreCoordinator *coordinator = [self persistentStoreCoordinator];
    if (!coordinator) {
        return nil;
    }
    _masterManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
    [_masterManagedObjectContext setPersistentStoreCoordinator:coordinator];
    [_masterManagedObjectContext setUndoManager:nil];
    return _masterManagedObjectContext;
}
-(NSManagedObjectContext*) backgroundManagedObjectContext{
    if(_backgroundManagedObjectContext != nil){
        return _backgroundManagedObjectContext;
    }
    _backgroundManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
    [_backgroundManagedObjectContext setUndoManager:nil];
    [_backgroundManagedObjectContext setParentContext:[self masterManagedObjectContext]];
    return _backgroundManagedObjectContext;
}
-(NSManagedObjectContext*) mainManagedObjectContext{
    if(_mainManagedObjectContext !=nil){
        return _mainManagedObjectContext;
    }
    _mainManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
    [_mainManagedObjectContext setUndoManager:nil];
    [_mainManagedObjectContext setParentContext:[self masterManagedObjectContext]];
    return _mainManagedObjectContext;
}

The import is processed in the backgroundManagedObjectContext. The Header is created and stored by using the below code:

_header = [NSEntityDescription insertNewObjectForEntityForName:@"Header" inManagedObjectContext:_backgroundManagedObjectContext];
_header.startTime = [NSDate date];
NSError *error;
BOOL success = [_backgroundManagedObjectContext save:&error];

The Received Data is created and stored by using the below code when the bluetooth devices fired the method:

@autoreleasepool {    
    ReceivedData* data = [NSEntityDescription insertNewObjectForEntityForName:@"ReceivedData" inManagedObjectContext:_backgroundManagedObjectContext];
    //Data is set here
    [_header addFk_header_many_dataObject:data];
    currentCount ++;
    if(currentCount >=1000){
        currentCount = 0;
        NSError *error;
        BOOL success = [_backgroundManagedObjectContext save:&error];
    }
}

the received data will be stored into the managedObjectContext per 1000 data is received.

Once if i stop the process, the memory consumed is doubled, and last until i completely terminate the app.

The code to handle the end of process is listed below:

_header.endTime = [NSDate date];
_header = nil;
NSError *error;
BOOL success = [_backgroundManagedObjectContext save:&error];
[_masterManagedObjectContext performBlock:^{
   NSError* mastererror;
   BOOL mastersuccess = [_masterManagedObjectContext save:&mastererror];
}];

Issue: As mentioned by Core Data Performance by Apple, using reset method of NSManagedObjectContext will remove all managed objects associated with a context and "start over" as if you'd just created it.

In my understanding, that means i can only call this method at the end of the whole process. I have tried to add reset function just after _backgroundManagedObjectContext and _masterManagedObjectContext is saved. However, the memory remain unchanged.

Illustration of the memory usage For the case of data is received on every 0.002 seconds, 0.5MB memory increased per 1000 records is saved to backgroundManagedObjectContext. Therefore, the app will consume around 150 MB for 8 mins process time and memory will increase to 320MB when the process terminated at that time, and will retain the memory usage around 220MB.

Questions: How do i release the memory used by the NSManagedObjectContext(i guess) when the number of records to be inserted to Core Data are unforeseeable, such that the memory can be efficiently used?

Sorry for some idiot mistakes as i am quite new in iOS. I have tried my best to search around before posting the question.

Your help is appreciated. Thank you very much.

Remarks I have tried the above mentioned case in no more than 10 mins process time. However, the implementation should have extended to the case for more than 1 hour process time. I still have no idea on the way of handling such case.

EDIT 1 modified the code for showing the relationship of ReceivedData and Header EDIT 2 updated the code for the standard mentioned by @flexaddicted

Answer 1

Just my advice. Maybe someone could have a different approach.

In this case I would eliminate the BackgroundManagedObjectContext and I would leave only the MasterManagedObjectContext (as the parent of the main one). Since, you need a low memory profile, you should switch to mechanism that allows you to control the memory footprint of your application. So, I would created a sort of buffer that starts to collect the receive data. When the buffer has reached its limit, I would move the receive data to the MasterManagedObjectContext in order to save them into the persistent store. Here the limit of the buffer (a vector of structs or array of objects in my mind) should be tuned on the application performance. In this way you have a direct control of the objects created. So, you can throw them away whenever you have finished a bunch of imported data (where the bunch is the limit of that vector/array).

Otherwise, you can try the following approach.

@autoreleasepool {    

    NSMutableArray *temporary = [NSMutableArray array];

    ReceivedData* data = [NSEntityDescription insertNewObjectForEntityForName:@"ReceivedData" inManagedObjectContext:_backgroundManagedObjectContext];
    // Data is set here

    // Let temporary to hold a reference of the data object 
    [temporary addObject:data];
    currentCount ++;
    if(currentCount >=1000){
        currentCount = 0;
        NSError *error;
        BOOL success = [_backgroundManagedObjectContext save:&error];

        for(NSManagedObject *object in temporary) {
            [_backgroundManagedObjectContext refreshObject:object mergeChanges:NO];
        }
        [temporary removeAllObjects];
    }
}

Update 1

Can you also show where you set the relationship between ReceiveData and Header? I'm asking this because you can change the time where you set the relationship between this two entity.

Based on your modified code.

@autoreleasepool {    
    receivedData* data = [NSEntityDescription insertNewObjectForEntityForName:@"ReceivedData" inManagedObjectContext:_backgroundManagedObjectContext];
    //Data is set here

    [_header addFk_header_many_dataObject:data];
    currentCount ++;
    if(currentCount >=1000){
        currentCount = 0;
        NSError *error;
        BOOL success = [_backgroundManagedObjectContext save:&error];
    }
}

If you are able to posticipate this association on the master queue (I guess you need to set the attribute as an optional one), you can do like the following:

@autoreleasepool {    
    ReceivedData* data = [NSEntityDescription insertNewObjectForEntityForName:@"ReceivedData" inManagedObjectContext:_backgroundManagedObjectContext];
    // Data is set here

    // Move it later
    //[_header addFk_header_many_dataObject:data];

    currentCount ++;
    if(currentCount >=1000){
        currentCount = 0;
        NSError *error;
        BOOL success = [_backgroundManagedObjectContext save:&error];
        [_backgroundManagedObjectContext reset];
    }
}

P.S. receiveData *data = ... should be ReceiveData *data = .... In other words, classes should start with a capital letter.