Search code examples
couchbasecouchbase-sync-gateway

Growing number of couchbase binary documents in XDCR destination bucket


I am running couchbase enterprise edition version 6.6.2 on Windows server 2016 standard edition. I have two buckets called A and B. Bucket A is configured to run with enable_shared_bucket_access = true, my sync gateway creates new documents in bucket A, a bunch of services change and delete these documents.

XDCR replicates documents from bucket A to bucket B. All changes to documents in bucket A are replicated to bucket B, except deletions in bucket A are not replicated to bucket B. When documents in bucket B get older than 62 days they get deleted by an external service.

Over time I noticed that 93% of the documents in bucket B are binary documents! My own documents are in JSON, I don’t use any kind of binary documents in my solution. This leads me to the conclusion that these binary documents are some internal couchbase documents.

Here is a example of these binary documents

{
        "$1": {
            "cas": 1667520921496387584,
            "expiration": 0,
            "flags": 50331648,
            "id": "_sync:rev:00001abd-1f99-4b4e-a695-d11574ea9ed8:0:",
            "type": "base64"
        },
        "pa": "<binary (1 b)>"
    },
    {
        "$1": {
            "cas": 1667484959445614592,
            "expiration": 0,
            "flags": 50331648,
            "id": "_sync:rev:00001abd-1f99-4b4e-a695-d11574ea9ed8:34:2-d3fb2d58672f853d98ce343d3ae84c1d",
            "type": "base64"
        },
        "pa": "<binary (1129 b)>"
}

My issue with these documents is that they increase dramatically over time! and they don’t get cleaned up automatically! So they just grow and consume resources!

  • What are these documents used for?
  • Why aren’t these documents cleaned automatically?
  • Is it safe to simply delete these documents?
  • Is this a bug or a feature? :-)

Regards, Siraf


Solution

  • The issue was solved by adding this AND NOT REGEXP_CONTAINS(META().id,"^_sync:rev") to the XDCR replication filter expression. This stopped binary documents do be replicated from bucket A to B.