Search code examples
amazon-web-servicesamazon-ec2aws-code-deploy

AWS CodeDeploy fails: Script at specified location run as user ubuntu failed with exit code 1

If I try to run deployment from a GitHub project to EC2 Instance using AWS CodeDeploy, I get these error messages from /var/log/aws/codedeploy-agent/codedeploy-agent.log

2022-08-02 08:33:57 INFO  [codedeploy-agent(5871)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb.
2022-08-02 08:33:57 INFO  [codedeploy-agent(5871)]: [Aws::CodeDeployCommand::Client 200 0.029191 0 retries] get_deployment_specification(deployment_execution_id:"CodeDeploy/us-east-2/prod/orpheus:public005/020801435925:d-GW7F45ARH",host_identifier:"arn:aws:ec2:us-east-2:020801435925:instance/i-087157ab53b339170")  

2022-08-02 08:33:58 INFO  [codedeploy-agent(5871)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb.
2022-08-02 08:33:58 INFO  [codedeploy-agent(5871)]: [Aws::CodeDeployCommand::Client 200 0.048105 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":4,\"script_name\":\"scripts/application_stop.sh\",\"message\":\"Script at specified location: scripts/application_stop.sh run as user ubuntu failed with exit code 1\",\"log\":\"LifecycleEvent - ApplicationStop\\nScript - scripts/application_stop.sh\\n[stderr][PM2][ERROR] Process or Namespace pm2stage not found\\n[stderr][PM2][ERROR] Process or Namespace pm2stage not found\\n\"}"},host_command_identifier:"eyJiYXRjaElkIjoiYTQ4MmNlZTY3OTZhM2M4NjcwODIxZjIwNjVkNjNmZDgvcHVibGljMDA1IiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTIvcHJvZC9vcnBoZXVzOnB1YmxpYzAwNS8wMjA4MDE0MzU5MjU6ZC1HVzdGNDVBUkgiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTI6MDIwODAxNDM1OTI1Omluc3RhbmNlL2ktMDg3MTU3YWI1M2IzMzkxNzAiLCJjb21tYW5kSWQiOiJBcG9sbG9EZXBsb3lDb250cm9sU2VydmljZXxhcm46YXdzOmVjMjp1cy1lYXN0LTI6MDIwODAxNDM1OTI1Omluc3RhbmNlL2ktMDg3MTU3YWI1M2IzMzkxNzB8MXwwIiwiY29tbWFuZE5hbWUiOiJBcHBsaWNhdGlvblN0b3AiLCJjb21tYW5kSW5kZXgiOjEsImF0dGVtcHRJbmRleCI6MX0=")  

2022-08-02 08:33:58 ERROR [codedeploy-agent(5871)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: InstanceAgent::Plugins::CodeDeployPlugin::ScriptError - Script at specified location: scripts/application_stop.sh run as user ubuntu failed with exit code 1 - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:183:in `execute_script'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:129:in `block (2 levels) in execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:117:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:117:in `block in execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:208:in `create_script_log_file_if_needed'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:115:in `execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:160:in `block (3 levels) in map'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:148:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:148:in `block (2 levels) in map'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:70:in `execute_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:115:in `process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:97:in `acknowledge_and_process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:76:in `block in perform'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
/opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2022-08-02 08:33:58 WARN  [codedeploy-agent(5871)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PutHostCommandComplete: "Code Error" 
2022-08-02 08:33:58 INFO  [codedeploy-agent(5871)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb.
2022-08-02 08:33:58 INFO  [codedeploy-agent(5871)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb.
2022-08-02 08:33:58 INFO  [codedeploy-agent(5871)]: [Aws::CodeDeployCommand::Client 200 0.026917 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"Script at specified location: scripts/application_stop.sh run as user ubuntu failed with exit code 1\",\"log\":\"\"}"},host_command_identifier:"eyJiYXRjaElkIjoiYTQ4MmNlZTY3OTZhM2M4NjcwODIxZjIwNjVkNjNmZDgvcHVibGljMDA1IiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTIvcHJvZC9vcnBoZXVzOnB1YmxpYzAwNS8wMjA4MDE0MzU5MjU6ZC1HVzdGNDVBUkgiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTI6MDIwODAxNDM1OTI1Omluc3RhbmNlL2ktMDg3MTU3YWI1M2IzMzkxNzAiLCJjb21tYW5kSWQiOiJBcG9sbG9EZXBsb3lDb250cm9sU2VydmljZXxhcm46YXdzOmVjMjp1cy1lYXN0LTI6MDIwODAxNDM1OTI1Omluc3RhbmNlL2ktMDg3MTU3YWI1M2IzMzkxNzB8MXwwIiwiY29tbWFuZE5hbWUiOiJBcHBsaWNhdGlvblN0b3AiLCJjb21tYW5kSW5kZXgiOjEsImF0dGVtcHRJbmRleCI6MX0=")  

2022-08-02 08:34:43 INFO  [codedeploy-agent(5871)]: [Aws::CodeDeployCommand::Client 200 45.11025 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-2:020801435925:instance/i-087157ab53b339170")  

2022-08-02 08:34:44 INFO  [codedeploy-agent(5871)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb.
2022-08-02 08:35:29 INFO  [codedeploy-agent(5871)]: [Aws::CodeDeployCommand::Client 200 45.141769 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-2:020801435925:instance/i-087157ab53b339170")  

2022-08-02 08:35:30 INFO  [codedeploy-agent(5871)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.3.2-1902_deb.
2022-08-02 08:36:16 INFO  [codedeploy-agent(5871)]: [Aws::CodeDeployCommand::Client 200 45.391621 0 retries] poll_host_command(host_identifier:"arn:aws:ec2:us-east-2:020801435925:instance/i-087157ab53b339170")

I don't understand what does it mean:

InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: InstanceAgent::Plugins::CodeDeployPlugin::ScriptError - Script at specified location: scripts/application_stop.sh run as user ubuntu failed with exit code 1

appspec.yml

version: 0.0
os: linux
files:
  - source: /
    destination: /var/apps/cbl_stage/cbl-backend
hooks:
  BeforeInstall:
    - location: scripts/before_install.sh
      timeout: 300
      runas: ubuntu
  AfterInstall:
    - location: scripts/after_install.sh
      timeout: 300
      runas: ubuntu
  ApplicationStart:
    - location: scripts/application_start.sh
      timeout: 300
      runas: ubuntu
  ApplicationStop:
    - location: scripts/application_stop.sh
      timeout: 300
      runas: ubuntu

before_install.sh

#!/bin/bash

#download node and npm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
. ~/.nvm/nvm.sh
nvm install 16.16.0

#create our working directory if it doesnt exist
DIR="/var/apps/cbl_stage/cbl-backend"
if [ -d "$DIR" ]; then
  echo "${DIR} exists"
else
  echo "Creating ${DIR} directory"
  mkdir ${DIR}
fi

after_install.sh

#!/bin/bash

#give permission for everything in the express-app directory
sudo chmod -R 777 /var/apps/cbl_stage/cbl-backend

#add npm and node to path
export NVM_DIR="$HOME/.nvm" 
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # loads nvm   
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # loads nvm bash_completion (node is in path now)

#navigate into our working directory where we have all our github files
cd /var/apps/cbl_stage/cbl-backend

#install node modules
npm ci
npm install pm2 -g

#migrate database
npx sequelize-cli db:migrate --env test
npx sequelize-cli db:seed:all --env test

application_start.sh

#!/bin/bash

#add npm and node to path
export NVM_DIR="$HOME/.nvm" 
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # loads nvm   
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # loads nvm bash_completion (node is in path now)

#navigate into our working directory where we have all our github files
cd /var/apps/cbl_stage/cbl-backend

#start our node app in the background
pm2 start "NODE_ENV=test ./bin/www" --name pm2stage --watch

application_stop.sh

#!/bin/bash

#add npm and node to path
export NVM_DIR="$HOME/.nvm" 
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # loads nvm   
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # loads nvm bash_completion (node is in path now)

#navigate into our working directory where we have all our github files
cd /var/apps/cbl_stage/cbl-backend

pm2 stop pm2stage
pm2 delete pm2stage
Solution

  • Well. The line

    put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{"error_code":4,"script_name":"scripts/application_stop.sh","message":"Script at specified location: scripts/application_stop.sh run as user ubuntu failed with exit code 1","log":"LifecycleEvent - ApplicationStop\nScript - scripts/application_stop.sh\n[stderr][PM2][ERROR] Process or Namespace pm2stage not found\n[stderr][PM2][ERROR] Process or Namespace pm2stage not found\n"}"},.....)

    says it all. Either of these commands:

    pm2 stop pm2stage

pm2 delete pm2stage

    is failing simply because pm2stage doesn't exist. It's either already stopped or not present because of a previous legitimate CI/CD failure. There are several ways by which you can handle it:

    • Check for the process before stopping or deleting it.

    #!/bin/bash
pm2 describe appname > /dev/null
RUNNING=$?

if [ "${RUNNING}" -ne 0 ]; then
  pm2 start ./deploy/development.yml
else
  pm2 restart appname
fi;

    • Ignore the exit code errors

      1. Using --silent param:

        pm2 stop --silent pm2stage

pm2 delete --silent pm2stage

      2. Using || ::

        pm2 stop pm2stage || :

pm2 delete pm2stage || :

    • Ignore the error output in CodeDeploy console before deployment

      You can choose to ignore the error in the ApplicationStop lifecycle event and continue with deployment. Under ApplicationStop lifecycle event failure, check the box that says Don’t fail the deployment to an instance if this lifecycle event on the instance fails. See screenshot for reference:

    enter image description here