GCD blocks on private queue stop executing with large SQLite3 databases

Question

I have a serial queue created as follows:

self.myQueue = dispatch_queue_create("com.myDomain.myQueue", DISPATCH_QUEUE_SERIAL);

I have an sqlite3 database to store up my data. Instead of committing this data every time, each time I get a piece of data, I store it up in a buffer - in this case, a lower level CPP-based 'vector'.

I add blocks onto my private queue to write to the buffer. I use dispatch_async() to do this. After a pre-defined time interval I commit data to the DB from the buffer.

When the DBs are below 5MB, things work as expected - each data comes in, each is written to the buffer, after the time interval expires, the data is committed to the DB from the buffer.

When the DB size increases to above 5 MB (approx.), the blocks on the queue seem not to execute until the time interval for committing data to the DB. When this time is done, one record is written to the buffer. The next time, the time interval expires, data is committed to the DB and again only one record is written to the buffer - this time a different record which is in the sequence i expect the records to come in. It seems like each block is executed only after I have written to the DB which happens only each time I am at the defined time interval.

Note: I have checked that these blocks that do execute, execute cleanly - without errors.

I cant understand the link here between the size of the DB and the non-execution (if there is such a word) of the blocks on the private queue.

Does thread width of he private queue have anything to do with it? Or memory concerns? I would say the size of the DB, but it seems like the every-time-interval-blocks execute correctly. Also, I would say DB size directly, but I'm not able to write to the buffer itself.

Any thoughts to give me direction OR is this an OS issue out of my control?

Answer 1

It's not quite clear how you've got it set up: It sounds like you've got a vector that's being protected for concurrent access using a serial queue. Then it sounds like you're also doing a periodic database operation on the same queue. Since it'a serial queue, if the database operations start to take longer and longer, the other operations are going to back up behind the database operation. I would recommend using the queue only for primitive operations on the vector, and using a second queue to manage your database operations. It might look like something like this:

using namespace std;

dispatch_queue_t vectorQueue = NULL;
dispatch_queue_t dbQueue = NULL;
dispatch_source_t timer = NULL;
vector<whatever>* dbRecords = NULL;

void init()
{
    dbRecords = new vector<whatever>();
    vectorQueue = dispatch_queue_create(NULL, DISPATCH_QUEUE_SERIAL);
    dbQueue = dispatch_queue_create(NULL, DISPATCH_QUEUE_SERIAL);
    timer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, dbQueue);
    dispatch_source_set_timer(timer, DISPATCH_TIME_NOW, NSEC_PER_SEC * 60, NSEC_PER_SEC);
    dispatch_source_set_event_handler(timer, ^{
        __block vector<whatever>* toCommit = NULL;
        dispatch_sync(vectorQueue, ^{
            if (dbRecords->size())
            {
                toCommit = new vector<whatever>(*dbRecords);
                dbRecords->clear();
            }
        });

        if (toCommit)
        {
            // Commit the records using your db code...

            delete toCommit;
        }
    });
    dispatch_resume(timer);
}

void enqueueRecord(whatever record)
{
    dispatch_async(vectorQueue, ^{
        dbRecords->push_back(record);
    });
}

This way, the incoming record blocks aren't backed up waiting for the database operation. The only blocks being executed on vectorQueue are operations on the shared vector. When its time to commit to the db, you copy out the records from the shared vector into a private copy, clear out the shared vector, and ship the private copy over to dbQueue to be committed.