Search code examples
pythondatabaselmdb

Maximum LMDB value size

Is there a maximum size of value that can be inserted into an LMDB database? I am trying to insert a 16GB file into an LMDB which is initialized to size 32GB and I obtain this error:

File "build_lmdb.py", line 90, in write_entry
    txn.put(key.encode('ascii'), value)
lmdb.BadValsizeError: mdb_put: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size

Is there any way around this, or any setup parameter that needs to be changed?

Solution

  • Yes, according to the MDB_val section of the LMDB documentation, the maximum size of a data item that can be entered into a database is 0xFFFF_FFFF bytes (which is 2^32 - 1 or about 4GiB), so 16GB/32GB would be too big for a single data item.

    (It looks like you're using the Python bindings, but since those are just bindings for the C APIs, whatever limitations apply to the C APIs should also be relevant).