How do I prevent swarm containers from becoming orphans upon removing the stack?

I run a Docker Swarm instance with the following restart script:

#!/usr/bin/env sh
docker stack rm owlab
sleep 10
docker stack deploy --compose-file ./docker-compose.yml owlab

docker-compose.yml:

version: "3"
services:
 webapp-front:
  image: "preprod.thatsowl.com:4200/webapp-front-dev"
  ports:
   - "80:80"
  volumes:
   - "../../webapp/frontend:/usr/src/app/"
 webapp-back:
  image: "webapp-back-dev"
  ports:
   - "4000:4000"
  volumes:
   - ../../webapp/backend/src:/usr/src/app/src
   - ../../webapp/backend/uploads:/usr/src/app/uploads
   - ../../webapp/shared:/usr/src/app/shared
   - ../../webapp/backend/html-minifier.conf:/usr/src/app/html-minifier.conf
  environment:
   - HOST_TO=http://localhost
   - DB_TO=local
  depends_on:
    - mongo
 mongo:
   image: mongo:4.2.8
   restart: always
   ports:
    - 27017:27017
   environment:
     MONGO_INITDB_ROOT_USERNAME: root
     MONGO_INITDB_ROOT_PASSWORD: example
   volumes:
    - ~/owlab_volumes/mongo:/data/db
 mongo-express:
   image: mongo-express
   restart: always
   ports:
    - 7081:8081
   environment:
    ME_CONFIG_MONGODB_ADMINUSERNAME: root
    ME_CONFIG_MONGODB_ADMINPASSWORD: example

Sometimes, when when I run my restart script, some containers stay alive. I would expect all containers to be removed upon removing the stack, as it usually does, but sometimes 1 or more containers stay alive, as if they decided to declare their independence. The frequency at which this happens and the number of containers involved when it happens are apparently random.

➜  local git:(master) ✗ docker service ls
ID        NAME      MODE      REPLICAS   IMAGE     PORTS
➜  local git:(master) ✗ docker ps
CONTAINER ID   IMAGE                                               COMMAND                  CREATED       STATUS       PORTS       NAMES
3a75b1930778   mongo:4.2.8                                         "docker-entrypoint.s…"   2 hours ago   Up 2 hours   27017/tcp   owlab_mongo.1.q7np7im4sbpbtdxe0l1q989dk
d086085894d4   3e0babb28f48                                        "npm run dev"            2 hours ago   Up 2 hours   4000/tcp    owlab_webapp-back.1.ykpkjb79tjq21dbr3fmhjoa21
58d178bba35f   preprod.thatsowl.com:4200/webapp-front-dev:latest   "npm start"              2 hours ago   Up 2 hours   80/tcp      owlab_webapp-front.1.jam4w1z3py8m52msrgx8k23hc

I tried to stop the container by hand but it freezes and the container stays alive forever.

➜  local git:(master) ✗ docker container stop 3a75b1930778

I also tried to use an image which I thought might solve my problem, but it freezes too:

➜  local git:(master) ✗ docker run --rm -v /var/run/docker/swarm/control.sock:/var/run/swarmd.sock dperny/tasknuke 3a75b1930778
Unable to find image 'dperny/tasknuke:latest' locally
latest: Pulling from dperny/tasknuke
88286f41530e: Pull complete 
0e61a138cf9f: Pull complete 
Digest: sha256:9e2e81971d201cee98f595f4516793333a4eb21bb9d7f7ca858ad2edb50353ad
Status: Downloaded newer image for dperny/tasknuke:latest
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

Note that this problem prevents me from creating my stack again:

➜  local git:(master) ✗ ./restart.sh
Removing network owlab_default
Failed to remove network iykl6khzqe646fra5xvddf9ow: Error response from daemon: network iykl6khzqe646fra5xvddf9ow not foundFailed to remove some resources from stack: owlab

Creating service owlab_webapp-front
failed to create service owlab_webapp-front: Error response from daemon: network owlab_default not found

Why are some containers randomly (as it seems) staying alive upon removing the stack? How do I prevent this from happening?

Solution

The deployment and removal of a stack is performed asynchronous. What you experience is high likely a race condition.

Make sure to wait until the objects from the removed stack are gone, before starting it again. I had such race conditions in the past as well.. this approach did the trick for me:

stack=owlab
docker stack rm ${stack}
types="service network config secret";
for type in $types;do 
  until [ -z "$(docker $type ls --filter label=com.docker.stack.namespace=${stack} -q)" ];do
    sleep 1
  done
done
docker stack deploy --compose-file ./docker-compose.yml ${stack}