By adding "graph": "/dcos/docker/" to /etc/docker/daemon.json , I changed the docker root dir for all my Mesosphere DCOS agnet-nodes to run under a larger disk instead of the root(/) volume.
Through docker info, I saw change from Docker Root Dir:/var/lib/docker/
to Docker Root Dir:/dcos/docker/.
Then from what i can see the change worked with Docker, since all volumes and all container are created in the directory of /dcos/docker. But the problem is that Mesosphere DCOS do not seem to notice such a change. When launching containers from the GUI or CLI, it can not start containers that needs a persistent volume larger than the size of root(/) even if the docker root volume is much larger.
/dev/mapper/centos-root 150G 4.3G 146G 3% /
/dev/sda1 881G 3.8G 832G 1% /dcos/docker
But as soon as i start a container that only needs a smaller size than the root disk, it works and places it under the /dcos/docker.
Is there any way to make Mesosphere DCOS recalculating the free space it can use for persistent storage? or make the agent-node to notice that the docker root dir is not under root(/) volume anymore?
In the gui, when asking looking at the debug "Recent Resource Offers" table, i can see that the node offerd 1T of storage but the task request 0 bytes and a red X on the disk paramter. But on other agent-nodes with larger roots than requested size, it is a green checkmark.
Got it working by changing the mount location of the disk from /dcos/docker to /var and changed back the location of docker root to /var/lib/docker. This fixed both root from filling up by docker containers and allows me to allocate larger presentient volumes in dcos.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 12G 39G 24% /
devtmpfs 148G 0 148G 0% /dev
tmpfs 148G 0 148G 0% /dev/shm
tmpfs 148G 250M 148G 1% /run
tmpfs 148G 0 148G 0% /sys/fs/cgroup
/dev/sda2 1014M 184M 831M 19% /boot
/dev/mapper/centos-lvol1 2.0T 39M 2.0T 1% /dcos/volume0
/dev/mapper/centos-lvol0 4.0T 17G 4.0T 1% /var
/dev/mapper/centos-lvol2 1.8T 33M 1.8T 1% /dcos/volume1
overlay 4.0T 17G 4.0T 1% /var/lib/docker/overlay/095f55a30c322914b0d4b82ac90a884fd5a1b1f937ce94654bf961e59644f720/merged
shm 64M 0 64M 0% /var/lib/docker/containers/e73e8277e9ce4270afd38038ecc2116e3a12e415783a611a74a7900f6241fe3a/shm
overlay 4.0T 17G 4.0T 1% /var/lib/docker/overlay/19c8656cf344a3dce3cbce73040961af551d38bb4f63843dd173d075c72ba58b/merged
shm 64M 0 64M 0% /var/lib/docker/containers/ccfcd9707003e55d229884ac283b977df634ae9d5c8b5f04517dba94c255fa5d/shm
overlay 4.0T 17G 4.0T 1% /var/lib/docker/overlay/886b3e427b2c2d8ff2854e015bd91617253c069fdf9537a882dac10584720924/merged
shm 64M 0 64M 0% /var/lib/docker/containers/654b4a4bd9aaa13720fbe3be81f48228d35932a45f5af56789c9fdd4b974294b/shm
tmpfs 30G 0 30G 0% /run/user/0