Search code examples
databasecouchdbfilesizepurge

why does CouchDBs _dbs.couch keep growing when purging/compacting DBs?

The setup:

  • A CouchDB 2.0 running in Docker on a Raspberry PI 3
  • A node-application that uses pouchdb, also in Docker on the same PI 3

The scenario:

  • At any given moment, the CouchDB has at max 4 Databases with a total of about 60 documents
  • the node application purges (using pouchdbs destroy) and recreates these databases periodically (some of them every two seconds, others every 15 minutes)
  • The databases are always recreated with the newest entries
  • The reason for purging the databases, instead of deleting their documents is, that i'd otherwise have a huge amount of deleted documents, and my web-client can't handle syncing all these deleted documents

The problem:

The file var/lib/couchdb/_dbs.couch always keeps growing, it never shrinks. Last time i left it alone for three weeks, and it grew to 37 GB. Fauxten showed, that the CouchDB only contains these up to 60 Documents, but this file still keeps growing, until it fills all the space available

What i tried:

  • running everything on an x86 machine (osx)
  • running couchdb without docker (because of this info)
  • using couchdb 2.1
  • running compaction manually (which didn't do anything)
  • googling for about 3 days now

Whatever i do, i always get the same result: the _dbs.couch keeps growing. I also wasn't really able to find out, what that files purpose is. googling that specific filename only yields two pages of search-results, none of which are specific.

The only thing i can currently do, is manually delete this file from time to time, and restart the docker-container, which does delete all my databases, but that is not a problem as the node-application recreates them soon after.

Solution

  • The _dbs database is a meta-database. It records the locations of all the shards of your clustered databases, but since it's a couchdb database too (though not a sharded one) it also needs compacting from time to time.

    try; curl localhost:5986/_dbs/_compact -XPOST -Hcontent-type:application/json

    You can enable the compaction daemon to do this for you, and we enable it by default in the recent 2.1.0 release.

    add this to the end of your local.ini file and restart couchdb;

    [compactions] _default = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}]