QList crashes when size is large

Question

I am using a QList to store the data read from a SQL Table. The table has more than a million records. I need to get them in a list and then do some processing on the list.

QList<QVariantMap> list;

QString selectNewDB = QString("SELECT * FROM newDatabase.M106SRData");
QSqlQuery selectNewDBQuery = QSqlDatabase::database("CurrentDBConn").exec(selectNewDB);
while (selectNewDBQuery.next())
{
    QSqlRecord selectRec = selectNewDBQuery.record();
    QVariantMap varMap;
    QString key;
    QVariant value;
    for (int i=0; i < selectRec.count(); ++i)
    {
        key = selectRec.fieldName(i);
        value = selectRec.value(i);
        varMap.insert(key, value);
    }
    list << varMap;
}

I get "qvector.h, line 534: Out of memory" error.

The program crashes when the list reaches the size of <1197762 items>. I tried using reserve() but it didn't work. Is QList limited to a specific size?

Answer 1

You've ran out of memory because the C++ runtime has reported that it cannot allocate any more memory. It's not a problem with Qt containers. The containers are limited to 2^31-1 items due to the size of int the use for the index. You're nowhere near that.

At the very least:

1. Use a QVector instead of QList as it has much lower overhead for the QVariantMap element.

2. Attempt to reserve the space if the query allows it: this will almost halve the memory requirements!

3. Compile for a 64 bit target if you can.

QVector<QVariantMap> list;

QString selectNewDB = QString("SELECT * FROM newDatabase.M106SRData");
QSqlQuery selectNewDBQuery = QSqlDatabase::database("CurrentDBConn").exec(selectNewDB);
auto const size = selectNewDBQuery.size();
if (size > 0) list.reserve(size);
while (selectNewDBQuery.next())
{
    auto selectRec = selectNewDBQuery.record();
    QVariantMap varMap;
    for (int i=0; i < selectRec.count(); ++i)
    {
        auto const key = selectRec.fieldName(i);
        auto const value = selectRec.value(i);
        varMap.insert(key, value);
    }
    list.append(varMap);
}