NO_SPACE - enumStoreLimitReached - enum store address space used

I am trying to feed a Vespa index, but I face a NO_SPACE error:

Detail resultType=FATAL_ERRORexception='ReturnCode(NO_SPACE, Put operation rejected for document 'id:site-search:site::<urn:uuid:033689d2-5f23-4afb-815d-d6f5d6ecbcbd>' of type 'site': 'enumStoreLimitReached: { action: "add more content nodes", reason: "enum store address space used (0.92813) > limit (0.9)", enumStore: { used: 31890298144, dead: 0, limit: 34359738368}, attributeName: "text", subdb: "ready"}')' endpoint=vespa1:8080 ssl=false resultTimeLocally=1532685239428

My Vespa nodes are running through Docker with volumes on the machine. Here are the volumes used:

-v /data/1/bench/vespa/sample-apps:/vespa-sample-apps -v /data/1/bench/vespa/logs:/opt/vespa/logs --volume /data/1/bench/vespa/var:/opt/vespa/var

So, if I'm right, the data should be stored in the /data/1 directory, and according to df:

$ df
Filesystem                 1K-blocks     Used  Available Use% Mounted on
udev                        65802860        0   65802860   0% /dev
tmpfs                       13162916   108488   13054428   1% /run
/dev/mapper/VgRoot-root     78217900  3249540   70952028   5% /
/dev/mapper/VgRoot-boot       237861    36998     184275  17% /boot
/dev/mapper/VgRoot-srv       9775612    36888    9222424   1% /srv
/dev/mapper/VgData01-data 3688163424 48757376 3451987424   2% /data/1

There is still a lot of space left there. So I was wondering, if maybe Vespa was not looking at the right disk partition to check if it is using more than 90% of the disk space.

Solution

There is an upper limit to the number of unique values for attributes per node - please refer to https://docs.vespa.ai/documentation/performance/attribute-memory-usage.html#data

The only remedy is to add more content nodes to the Vespa application - this is auto-redistribute documents over more nodes and hence reduce number of unique values in the attribute