Insert image in blob in Sqlite

Question

I am reading a directory for 600,000 images and wanting to store these images in a Sqlite DB. The DB structure is simply ID, IMAGE (blob).

I'm not proficient in C++ so am figuring this out.

First I open the DB file and setup the prepare statement etc

int rc = sqlite3_open_v2(filename, &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL);

char* errorMessage;
sqlite3_exec(db, "PRAGMA synchronous=OFF", NULL, NULL, &errorMessage);
sqlite3_exec(db, "PRAGMA count_changes=OFF", NULL, NULL, &errorMessage);
sqlite3_exec(db, "PRAGMA journal_mode=MEMORY", NULL, NULL, &errorMessage);
sqlite3_exec(db, "PRAGMA temp_store=MEMORY", NULL, NULL, &errorMessage);
sqlite3_exec(db, "PRAGMA cache_size=1", NULL, NULL, &errorMessage);

sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errorMessage);

char const *szSQL = "INSERT INTO images (image) VALUES (?);";
int rc = sqlite3_prepare_v2(db, szSQL, -1, &stmt, NULL);
if( rc != SQLITE_OK ) {
  printf("PREPARE FAILED. EXITING\n");
  exit(0);
}

There is then a while loop iterating over all the files in the directory. To read the image into a char* (for the blob) I use the following code:

char text[10] = {'\0'};
struct stat s;
int status = stat(fullimagepath.c_str(), &s);
int fd = open(fullimagepath.c_str(), O_RDONLY);
if (fd == -1)
{
    perror("Error opening file for reading");
    exit(1);
}

char *fileContent = (char *)mmap(NULL, s.st_size, PROT_READ, MAP_SHARED, fd, 0);
close(fd);

The code below is run on each iteration as well where it binds the blob and attempts the insert

int retVal = sqlite3_bind_blob(stmt, 1, fileContent, s.st_size, NULL);
if (retVal != SQLITE_OK) {
  fprintf(stderr, "ERROR %s %s\n", fullimagepath.c_str(), sqlite3_errmsg(db));
  sqlite3_finalize(stmt);
  sqlite3_close(db);
  exit(1);
}

retVal = sqlite3_step(stmt);
if (retVal != SQLITE_DONE && retVal != SQLITE_ROW) {
      sqlite3_finalize(stmt);
      sqlite3_close(db);
      fprintf(stderr, "ERROR %d\n", retVal);
      exit(1);
}
fprintf(stderr, "1f2 %s\n", fullimagepath.c_str());


sqlite3_reset(stmt);
sqlite3_clear_bindings(stmt);

Where fullimagepath is the full path to the image each time (the above is in a while loop iterating through the files in the directory).

Once the full directory is done we run the commit and close the DB.

sqlite3_exec(db, "COMMIT", NULL, NULL, &errorMessage);
sqlite3_finalize(stmt);
sqlite3_close(db);

The issue I am having is that after running for a number of the files it stops with a segmentation fault.

As far as I can tell through various trials where I changed the blob to text and other tests, this only happens when trying to insert the blob.

Can anyone point me in the right direction to fix this please?

When running on a smaller directory with less than 10,000 images it works fine. Or if you need more information please advise.

Thanks.

Answer 1

I finally figured out the issue after @Surt in the very first comment mentiond "where do you munmap". I didn't know what this meant, and after much searching in other places noticed a comment mentioning this. A little more playing and realised this is quite crucial. Placed the following

status = munmap(fileContent, s.st_size);
sqlite3_reset(stmt);

Thanks for those who commented for the pointers in the right direction. Much appreciated.