Search code examples
dockerdocker-swarm

Orphaned Tasks in Docker Swarm after removal of failed node

Last week I had to remove a failed node from my Docker Swarm Cluster, leaving some tasks that ran on that node in desired state "Remove".

Even after deleting the stack and recreating it with the same name, docker stack ps stackname still shows them.

Interestingly enough, after recreating the stack, the tasks are still there, but with no node assigned. screenshot of docker stack ps

Here's what I tried so far to "cleanup" the stack:

  • Recreating the stack with the same name
  • docker container prune
  • docker volume prune
  • docker system prune

Is there a way to remove a specific task?

Here's the output for docker inspect fkgz0oihexzs, the first task in the list:

[
  {
    "ID": "fkgz0oihexzsjqwv4ju0szorh",
    "Version": {
      "Index": 14422171
    },
    "CreatedAt": "2018-11-05T16:15:31.528933998Z",
    "UpdatedAt": "2018-11-05T16:27:07.422368364Z",
    "Labels": {},
    "Spec": {
      "ContainerSpec": {
        "Image": "redacted",
        "Labels": {
          "com.docker.stack.namespace": "redacted"
        },
        "Env": [
          "redacted"
        ],
        "Privileges": {
          "CredentialSpec": null,
          "SELinuxContext": null
        },
        "Isolation": "default"
      },
      "Resources": {},
      "Placement": {
        "Platforms": [
          {
            "Architecture": "amd64",
            "OS": "linux"
          }
        ]
      },
      "Networks": [
        {
          "Target": "3i998stqemnevzgiqw3ndik4f",
          "Aliases": [
            "redacted"
          ]
        }
      ],
      "ForceUpdate": 0
    },
    "ServiceID": "g3vk9tgfibmcigmf67ik7uhj6",
    "Slot": 1,
    "Status": {
      "Timestamp": "2018-11-05T16:15:31.528892467Z",
      "State": "new",
      "Message": "created",
      "PortStatus": {}
    },
    "DesiredState": "remove"
  }
]
Solution

  • I had the same problem. I resolved it following this instructions :

    docker run --rm -v /var/run/docker/swarm/control.sock:/var/run/swarmd.sock dperny/tasknuke <taskid>

    Be sure to use the full long task id or it will not work (fkgz0oihexzsjqwv4ju0szorh in your case).