Search code examples
databasetime-seriesquestdb

Table busy error when reading from QuestDB concurrently

We are using QuestDB and reading data from Python. It behaves well generally, but we get table busy errors whenever we are launching many queries at the same time.

psycopg2.DatabaseError: [-1]: table busy [reason=unknown]

If we are querying only a few queries in parallel is all good, but when we start sending a few hundreds of concurrent queries we are getting the error.

The query shouldn't be blocking anything as it is just reading data. Also, just in case I checked the TX WAL and the table is caught up.

Solution

  • This can happen when we run out of pooled readers. QuestDB uses a default of 10*16 readers. It can be increased by changing:

    cairo.reader.pool.max.segments=10

    Or the equivalent as an env variable

    QDB_CAIRO_READER_POOL_MAX_SEGMENTS=10

    Switching to some number N such as N*16 is below our expected query concurrency should help here.