We are configuring a CodePipeline with CodeDeploy and CodeCommit. I've configured our CodeCommit repo and can successfully deploy to an EC2 instance that is not in an AutoScaling Group. However, when I create an AMI from that same instance and use it to build a launch template and Auto Scaling group, the instances in that ASG fail to download the bundle. I've verified that the IAM role assigned to my EC2 instance is also assigned to my ASG launch template. I'll add that our ASG instances are on a private subnet with no NAT gateway. We have followed AWS' instructions for using using CodeDeploy with Amazon Virtual Private Cloud and a VPC endpoint configured, I believe correctly. (The ASG instances see port 443 as open on the CodeDeploy servers.)
Before the DownloadBundle step fails, BeforeBlockTraffic, BlockTraffic, AfterBlockTraffic, and ApplicationStop all succeed.
The DownloadBundle fails, though. The AWS console just reports "UnknownError". The codedeploy-agent.log on the ASG instance, reports:
2023-06-26T17:56:11 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Executing DownloadBundle command for execution CodeDeploy/us-east-2/prod/orpheus:public005/244301334150:d-G64NUIVVN 2023-06-26T17:56:11 INFO [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Downloading artifact bundle from bucket 'codepipeline-us-east-2-457949292945' and key 'clc-pl-2/SourceArti/yOCp19T', version '', etag 'e27418ca132ed2ca7a7c4e8fab91472a-1' 2023-06-26T17:56:12 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand: 2023-06-26T17:56:12 INFO [codedeploy-agent(3909)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.6.0-49_rpm. 2023-06-26T17:56:57 INFO [codedeploy-agent(3909)]: [Aws::CodeDeployCommand::Client 200 45.197855 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-2:244301334150:instance/i-01bb942fef9c3eda3")
2023-06-26T17:56:57 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: PollHostCommand: Host Command = nil 2023-06-26T17:56:58 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand: 2023-06-26T17:56:58 INFO [codedeploy-agent(3909)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.6.0-49_rpm. 2023-06-26T17:57:13 DEBUG [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PutHostCommandComplete: "Code Error" 2023-06-26T17:57:13 INFO [codedeploy-agent(3909)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.6.0-49_rpm. 2023-06-26T17:57:13 INFO [codedeploy-agent(3909)]: [Aws::CodeDeployCommand::Client 200 0.048875 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{"error_code":5,"script_name":"","message":"execution expired","log":""}"},host_command_identifier:"eyJiYXRjaElkIjoiZGE4YTM4NmRlYmI3NjljOTcyMWU5Zjk0YzllYTUzMjQvcHVibGljMDAzIiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTIvcHJvZC9vcnBoZXVzOnB1YmxpYzAwNS8yNDQzMDEzMzQxNTA6ZC1HNjROVUlWVk4iLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTI6MjQ0MzAxMzM0MTUwOmluc3RhbmNlL2ktMDFiYjk0MmZlZjljM2VkYTMiLCJjb21tYW5kSWQiOiJBcG9sbG9EZXBsb3lDb250cm9sU2VydmljZXxhcm46YXdzOmVjMjp1cy1lYXN0LTI6MjQ0MzAxMzM0MTUwOmluc3RhbmNlL2ktMDFiYjk0MmZlZjljM2VkYTN8NXwwIiwiY29tbWFuZE5hbWUiOiJEb3dubG9hZEJ1bmRsZSIsImNvbW1hbmRJbmRleCI6NSwiYXR0ZW1wdEluZGV4IjoxfQ==")
2023-06-26T17:57:13 ERROR [codedeploy-agent(3909)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Seahorse::Client::NetworkingError - execution expired - /usr/share/ruby/net/http.rb:937:in `initialize'
I've looked over my configuration many times and can't find what is wrong. Any ideas?
The problem was that I had not yet created an S3 gateway endpoint. Even though I had the CodeDeploy endpoints, I was missing the S3 gateway endpoint. Because my instance is in a private subnet and there is no NAT instance/gateway configured, the instance needed the S3 gateway endpoint to communicate with AWS S3 service.
After following these instructions things started working: https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html