Search code examples
amazon-web-servicesdockeramazon-ec2docker-machine

"docker-machine rm" failing on non-existent EC2 instance

I use docker-machine to manage Docker nodes on AWS. Today I tried to create a new node and failed due to an incompatibility between the instance type and specified AMI:

docker-machine create --driver amazonec2 --amazonec2-instance-type t2.micro --amazonec2-ami ami-b4a015d4 certbot-config
Running pre-create checks...
Creating machine...
(certbot-config) Launching instance...
Error creating machine: 
    Error in driver during machine creation: 
    Error launching instance: InvalidParameterCombination: Virtualization type 'hvm' is required for instances of type 't2.micro'. Ensure that you are using an AMI with virtualization type 'hvm'. For more information, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html

Whatever, I correct the error. My problem is that I seem to have created a ghost machine; if I run docker-machine ls, I get this:

NAME             ACTIVE   DRIVER      STATE   URL   SWARM   DOCKER    ERRORS
certbot-config   -        amazonec2   Error                 Unknown   MissingParameter: The request must contain the parameter InstanceId
             status code: 400, request id:

I can't remove it, kill it, restart it, or provision it; every command exits with a 400 error from the AWS driver:

~$ docker-machine rm certbot-config
About to remove certbot-config
Are you sure? (y/n): y
Error removing host "certbot-config": unknown instance
MissingParameter: The request must contain the parameter KeyName
    status code: 400, request id:

How can I clear this dangling instance? I have many instances running under docker-machine, so I'd rather not do anything drastic, like reinstalling.

Edit:

Perhaps another way to frame this: where is docker-machine keeping track of managed hosts? Presumably there's a file somewhere in my computer with a list of hosts, and when I run "docker-machine ls" it goes down that list and pings for status. Where is this file, and can I safely delete hosts that I know to be dead without leaving garbage elsewhere?

Solution

  • Perhaps another way to frame this: where is docker-machine keeping track of managed hosts?

    Check if you have an environment variable MACHINE_STORAGE_PATH.

    If not, those VMS should be in ~/.docker.

    You can see also the unresolved issue 3555:

    Using docker-machine to provision a spot instance opens a spot request and waits for it to be fulfilled. If this fails, docker-machine exits but the spot request remains open and may be fulfilled later. This results in a dangling spot request with an active instance.

    Spot instance support was discussed for docker-machine in issue 226.

    From Spot Request, the main additional parameters seem to be Bid, Request valid From/To and Persistent request.

    http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/spot_introduction.png

    In the context of machine, I guess you could set request valid to = now + 2 minutes or so, at which point the command could fail.

    The PR adding the support was PR 942 and PR 1000.
    It is possible docker-machine rm does not adequately find/provide the extra parameters that a spot instance expects.

    In addition to removing the corresponding directory from ~/.docker/machine/machines, you need to remove the corresponding EC2 key pair from the AWS EC2 console.