We have an Ubuntu server that runs a few test environments under Docker. The system ran out of space today, and I eventually found that /var/lib/docker had grown to 826 GB with /var/lib/docker/volumes alone making 600 GB thereof.
Turns out that docker has created over 1,300 volumes:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 12 12 9.271GB 447.8MB (4%)
Containers 12 10 405.9MB 62.84MB (15%)
Local Volumes 1307 8 634.8GB 630.8GB (99%)
Build Cache 3586 0 19.05GB 19.05GB
I tried docker volume prune (and also with force-pruning images and containers before that), but that removed only a handful volumes.
We have another server with the same test environments under Docker, and it's looks totally different (i.e. normal) with only 9 volumes:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 13 13 10.07GB 447.8MB (4%)
Containers 14 10 312.2MB 111.7MB (35%)
Local Volumes 9 9 5.212GB 0B (0%)
Build Cache 0 0 0B 0B
If I check one of these volumes to find out which contain er or image it belongs to, I get nothing:
$ docker ps -a --filter volume=ff17641242297f3f9f3a63b2c328168f93c514e6e41cc3ed19bfbcb192acb5c7
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
The I tried a more radical solution:
$ docker system prune -a -f
It deleted a lot of build cache objects (21 GB in total), but no volumes.
How can it be that Docker has created 1,300 volumes that cannot be pruned? And what can I do to fix that?
Note that I found Why did my docker volume run out of disk space?, but the solutions given in the answers there do not work, so I assume my problem is a different one.
This command eventually did it:
docker volume rm $(docker volume ls -q --filter dangling=true)
According to the docs, docker volume prune -f should actually do the job, but it didn't. According to this answer, the above is "the old way" to do it, but obviously, it is the way it actually works, too.