Jenkins docker swarm volume gets permission denied

Question

I've got Jenkins running in a docker container, with swarm agents.

I've created a docker volume: 'maven-repository' and can get a job to access this (to cache maven). However, if two jobs run simultaneously I get a permission denied error, sample logs below:

Anyone else had this issue and managed to solve it? I've also tried mounting /var/jenkins_home/.m2:/tmp/.m2

Successful run:

Running on Jenkins in /var/jenkins_home/workspace/Alex Test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Agent Setup)
[Pipeline] sh
[Alex Test] Running shell script
+ docker pull maven:3.5.3-jdk-8-alpine
3.5.3-jdk-8-alpine: Pulling from library/maven
Digest: sha256:d8c6a5fef17ae7fcb2629e558554a085e90c722796306f31bee7fb7b9a5a123e
Status: Image is up to date for maven:3.5.3-jdk-8-alpine
[Pipeline] }
[Pipeline] // stage
[Pipeline] sh
[Alex Test] Running shell script
+ docker inspect -f . maven:3.5.3-jdk-8-alpine
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 73c7522e3a5de318e5500a6092974cc78ab5eedf4de70b18d264aeb40e40b360
$ docker run -t -d -u 1000:1000 --privileged -v maven-repository:/tmp/.m2 -v /var/run/docker.sock:/var/run/docker.sock -w "/var/jenkins_home/workspace/Alex Test" --volumes-from 73c7522e3a5de318e5500a6092974cc78ab5eedf4de70b18d264aeb40e40b360 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** maven:3.5.3-jdk-8-alpine cat
$ docker top 9902bb5b16a8b5a0b5ee16f2a0c87ea9961fe06c895b859f96142adf388be1b9 -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Check Maven Directory)
[Pipeline] sh
[Alex Test] Running shell script
+ ls /tmp
[Pipeline] sh
[Alex Test] Running shell script
+ ls /tmp/.m2
hsperfdata_root
repository
[Pipeline] sh
[Alex Test] Running shell script
+ cat /dev/urandom
+ fold -w 32
+ head -n 1
+ tr -dc a-zA-Z0-9
+ touch /tmp/.m2/68rS0lsIiml0cIg2bJ8BooxltUqJy31X
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 9902bb5b16a8b5a0b5ee16f2a0c87ea9961fe06c895b859f96142adf388be1b9
$ docker rm -f 9902bb5b16a8b5a0b5ee16f2a0c87ea9961fe06c895b859f96142adf388be1b9
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

If a Job is run at the same time:

[Pipeline] withDockerContainer
swarm-agent seems to be running inside container 9475a72289efb2b4a0918fa5127514e81bea82b90720dd859380fe9ffc1f4d92
$ docker run -t -d -u 10000:10000 --privileged -v maven-repository:/tmp/.m2 -v /var/run/docker.sock:/var/run/docker.sock -w "/home/jenkins/agent/workspace/Alex Test@3" --volumes-from 9475a72289efb2b4a0918fa5127514e81bea82b90720dd859380fe9ffc1f4d92 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** maven:3.5.3-jdk-8-alpine cat
$ docker top b7b9793e52ae9497209b7469c936e98d157d408ad5235776e26b3c05c2a151da -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Check Maven Directory)
[Pipeline] sh
[Alex Test@3] Running shell script
+ ls /tmp
[Pipeline] sh
[Alex Test@3] Running shell script
+ ls /tmp/.m2
[Pipeline] sh
[Alex Test@3] Running shell script
+ fold -w 32
+ cat /dev/urandom
+ tr -dc a-zA-Z0-9
+ head -n 1
+ touch /tmp/.m2/8lqaDHloPUqmw6a4Wr3aky7AZdDhMIrW
touch: /tmp/.m2/8lqaDHloPUqmw6a4Wr3aky7AZdDhMIrW: Permission denied
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 b7b9793e52ae9497209b7469c936e98d157d408ad5235776e26b3c05c2a151da
$ docker rm -f b7b9793e52ae9497209b7469c936e98d157d408ad5235776e26b3c05c2a151da
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

Answer 1

We found that the master was running our first job which has the docker volume, however running multiple jobs caused the slave to be used which did not have the docker volume present.