Cannot Delete ENI - RDS Already Deleted

Question

I tried to deploy code to an ElasticBeanstalk environment. Every-time I try to deploy this branch to an environment EB kills all instances, ELB, RDS, etc and tries to rebuild but fails. This leaves the environment in a bad state because it deletes the RDS but does not delete the security groups or ENI. When I try to delete the security groups manually it fails saying there are dependant objects.

I traced it back to the network interface but when I try to detach it (even force detach) I get an error that I do not have permission. This ENI should have been removed with the RDS instance but it was not. Now I cannot get rid of the environment at all and cannot rebuild it.

I am not sure why this application would cause the environment to attempt to re-create everything upon every deployment as the EC2 instances go away and then when they load back up they are added to the ELB however the ELB cannot do the health checks so they are constantly put Out of Service and the environment is in a dead state. It would be nice if I could somehow see the logs as to what is causing the environments to crash with this application.

Having ElasticBeanstalk delete all instances including RDS is not acceptable for a deployment because we constantly have to re-seed this, not to mention if this were ever deployed to production it would wipe all production data and we cannot have that.

Is there a way to see what is going on during a deployment and why this may be happening?

Answer 1

Elastic Beanstalk uses CloudFormation behind the scenes. You will be able to delete the entire environment by identifying the correct stacks (prefixed by awseb-e-j5zfptidfe-stack according to your logs) and removing them - or at least removing the one with the ENI.

You will also need to remove the environment from within ElasticBeanstalk. This will reset everything. If there are dependent stacks - like with the security groups. The best solution is to read the messages to determine the dependencies and clean those up first.

It is good practice to not include your RDS in the elastic beanstalk stack if you know you will want to preserve the data in it. Create this separately and just pass in the connection details to your stack. AWS provide detailed instructions. A short summary would be:

1. Create a security group for the database
2. Create an RDS database with the security group
3. Add the database connection paremeters as environment variables to your EB stack
4. Add the EC2 security group to your database security group as an allowed source of traffic to the db.

Finally. You need to determine why the instances are being terminated in your stack. It looks like they are not becoming 'healthy'. Disable Ignore health check which is an option for Elastic Beanstalk deployments.

This should result in an environment with EC2 instances marked 'unhealthy'. You can then use whatever tools you need to, to diagnose why the EC2 instances are not correctly responding to the health checks and resolve the issue.

There could be many reasons the EC2 instance fails the health check. The check itself could be incorrectly configured, security groups could be wrong or the service on the EC2 instance itself may not be responding as it should.