Still fairly new to AWS Codepipeline and I am trying to pass an output artifact out into the next stage of my build. In this particular case, I want to do two artifacts for the builds phase which is the only part I am focusing on right now. I have included my codepipeline code for reference:
resource "aws_codepipeline" "cp_plan_pipeline" {
name = "${local.name_prefix_H}-${var.cp_name}"
role_arn = aws_iam_role.cp_service_role.arn
artifact_store {
type = var.cp_artifact_type
location = module.S3.bucket_name
}
stage {
name = "Clone"
action {
name = "Source"
category = "Source"
owner = "AWS"
provider = "CodeCommit"
input_artifacts = []
version = "1"
output_artifacts = ["CodeWorkspace"]
configuration = {
RepositoryName = var.cp_repo_name
BranchName = var.cp_branch_name
PollForSourceChanges = var.cp_poll_sources
OutputArtifactFormat = var.cp_ouput_format
}
run_order = "1"
}
}
stage {
name = "Plan"
action {
name = "Terraform-Plan"
category = "Build"
owner = "AWS"
provider = "CodeBuild"
version = "1"
input_artifacts = ["CodeWorkspace"]
output_artifacts = ["CodeSource","TerraformPlanFile"]
configuration = {
ProjectName = var.cp_plan_project_name
EnvironmentVariables = jsonencode([
{
name = "PIPELINE_EXECUTION_ID"
value = "#{codepipeline.PipelineExecutionId}"
type = "PLAINTEXT"
}
])
}
}
}
stage {
name = "Test"
action {
name = "Testing"
category = "Test"
owner = "AWS"
provider = "CodeBuild"
input_artifacts = ["CodeSource"]
output_artifacts = ["TestOutput"]
version = "1"
configuration = {
ProjectName = var.cp_test_project_name
EnvironmentVariables = jsonencode([
{
name = "PIPELINE_EXECUTION_ID"
value = "#{codepipeline.PipelineExecutionId}"
type = "PLAINTEXT"
}
])
}
}
}
stage {
name = "Manual-Approval"
action {
run_order = 1
name = "AWS-Admin-Approval"
category = "Approval"
owner = "AWS"
provider = "Manual"
version = "1"
input_artifacts = []
output_artifacts = []
configuration = {
CustomData = "Please verify the terraform plan output on the Plan stage and only approve this step if you see expected changes!"
}
}
}
stage {
name = "Deploy"
action {
run_order = 1
name = "Terraform-Apply"
category = "Build"
owner = "AWS"
provider = "CodeBuild"
input_artifacts = ["TerraformPlanFile"]
output_artifacts = []
version = "1"
configuration = {
ProjectName = var.cp_apply_project_name
PrimarySource = "CodeWorkspace"
EnvironmentVariables = jsonencode([
{
name = "PIPELINE_EXECUTION_ID"
value = "#{codepipeline.PipelineExecutionId}"
type = "PLAINTEXT"
}
])
}
}
}
}
So the pipeline works fine as far as the deployment of the pipeline goes. My buildspec works great up until the point where I get to the artifact stage of the buildspec where i get the following error: Phase context status code: CLIENT_ERROR Message: no definition for secondary artifact CodeSource in buildspec My issue is the buildspec and trying to output the second source so i can upload it for the second phase. Please note some parts of the buildspec such as variables have been scrubbed. Buildspec below:
version: 0.2
env:
variables:
TF_VERSION: "1.0.7"
PY_VERSION: "3.9.6"
GIT_VERSION: "2.9.5"
PACKER_VERSION: "1.7.8"
JQ_VERSION: "1.6"
TFLINT_VERSION: "0.34.0"
PERMISSION_SETS_DIR: "CodeSource"
phases:
install:
commands:
# iNSTALL/UPDATE SSH CLIENT
- echo UPDATING SSH CLIENT
- "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"
# iNSTALL JQ JSON PARSER
- curl -s -qL -o jq https://github.com/stedolan/jq/releases/download/jq-${JQ_VERSION}/jq-linux64
- chmod +x ./jq
- cp jq /usr/bin
# INSTALL TERRAFORM
- echo STARTING TERRAFORM INSTALLATION
- curl -s -qL -o terraform.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip
- unzip terraform -d /usr/bin/
- "chmod +x /usr/bin/terraform"
- "/usr/bin/terraform --version"
# INSTALL Packer
- echo STARTING PACKER INSTALLATION
- curl -s -qL -o packer.zip https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip
- unzip packer -d /usr/bin/
- "chmod +x /usr/bin/packer"
- "/usr/bin/packer --version"
# INSTALL PYTHON
- echo "STARTING PYTHON INSTALLATION"
- "curl -s -qL -o python.tgz https://www.python.org/ftp/python/${PY_VERSION}/Python-${PY_VERSION}.tgz"
- "tar xf python.tgz -C /usr/bin/"
- "python --version"
- python -m pip install -U pip
# ADD PYTHON MODULES
- echo "INSTALL PYTHON MODULES"
- pip install git-remote-codecommit
pre_build:
commands:
# Adds a private SSH key to allow us to clone or npm install Git repositories
- rootpath=$(pwd)
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
# Configure SSH Key
- echo "$ssh_key" > ~/.ssh/cc_rsa
- cd ~/.ssh/
- cat cc_rsa
- |
echo "Multiline command"
cat > ~/.ssh/config <<EOL
Host idt-codecommit
Hostname git-codecommit.us-east-1.amazonaws.com
User ${cc_user}
IdentityFile ~/.ssh/cc_rsa
EOL
- cat ~/.ssh/config
# Configure SSH Permissions
- echo +++++++++CONFIG SSH+++++++++
- chmod 700 ~/.ssh
- chmod 600 ~/.ssh/config
- chmod 600 ~/.ssh/cc_rsa
- ssh-keyscan -t rsa1,rsa,dsa git-codecommit.us-east-1.amazonaws.com >> ~/.ssh/known_hosts
# Clone directories
- echo +++++++++CLONE DIRECTORIES+++++++++
- mkdir -p ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}
- cd ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}
- git clone codecommit://sourcerepo local_primary_repo
- git clone -b development ssh://sourcerepo2/v1/repos/sourcerepo2
# GET AWS ACCOUNT VARIABLES
- aws_region=$AWS_DEFAULT_REGION
- awsaccountnumber=$(aws sts get-caller-identity --query "Account")
- awsaccountname=$(aws iam list-account-aliases | jq -r '.AccountAliases | .[]')
- |
if [[ "$awsaccountname" == *"prod"* ]]; then
appenv="prod"
else
if [[ "$awsaccountname" == *"test"* ]]; then
appenv="test"
else
if [[ "$awsaccountname" == *"dev"* ]]; then
appenv="dev"
else
if [[ "$awsaccountname" == *"sbx"* ]]; then
appenv="sbx"
else
appenv="other"
fi
fi
fi
fi
build:
on-failure: ABORT
commands:
# Import S3 Folder Location Variable
- cd ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}/local_primary_repo
- CODEBUILD_GIT_BRANCH=`git symbolic-ref HEAD --short 2>/dev/null`
# CHECK TERRAFORM CODE
- cd ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}/local_primary_repo
- echo "yes" | terraform init
- terraform validate > ${bucketconfig}-tfvalidateexport.txt
- terraform validate
- terraform plan -out=tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID}
- cp ${bucketconfig}-tfvalidateexport.txt ${CODEBUILD_SRC_DIR}/
- cp tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID} ${CODEBUILD_SRC_DIR}/
post_build:
commands:
- echo "Terraform plan completed on `date`"
artifacts:
files:
- tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID}
name: TerraformPlanFile
secondary-artifacts:
artifact_1:
files:
- '**/*'
name: CodeSource
base-directory: ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}
Links or advice of what i am doing wrong would be very helpful. Thank you!!
First of I wanted to thank Antonio González for his post. It was exactly what I needed for the solution. I would like to recap the solution here for any other folks looking for an answer to the same issue. I have a working understanding that will help beginners. When Outputting multiple artifacts you have to use the secondary artifacts: action within the phase of the artifacts. See the example below as you follow along:
artifacts:
base-directory: ${CODEBUILD_SRC_DIR}/
files:
- '**/*'
secondary-artifacts:
ARTIFACT1:
base-directory: DIRECTORY_FOR_ARTIFACT_1/
files:
- FILES_YOU_LOOKING_TO_OUTPUT
name: ARTIFACT_1_NAME_ANYTHING_YOU_WANT
ARTIFACT2:
base-directory: DIRECTORY_FOR_ARTIFACT_2/
files:
- FILES_YOU_LOOKING_TO_OUTPUT
name: ARTIFACT_2_NAME_ANYTHING_YOU_WANT
So this is the basic setup for multiple artifacts, if you wanted to do another artifact you would do a third. Now the key to this setup is mapping the secondary-artifacts headings to the outputs that you defined in your codepipeline output. Recall my pipeline build phase:
stage {
name = "Plan"
action {
name = "Terraform-Plan"
category = "Build"
owner = "AWS"
provider = "CodeBuild"
version = "1"
input_artifacts = ["CodeWorkspace"]
output_artifacts = ["CodeSource","TerraformPlanFile"]
}
}
The output artifacts MUST be associated with the secondary-artifacts heading as shown below:
secondary-artifacts:
TerraformPlanFile:
base-directory: ${CODEBUILD_SRC_DIR}/
files:
- tfplan_commitid_${CODEBUILD_RESOLVED_SOURCE_VERSION}_pipelineid_${PIPELINE_EXECUTION_ID}
name: TerraformPlanFile
CodeSource:
base-directory: ${CODEBUILD_SRC_DIR}/configurations/${PERMISSION_SETS_DIR}/
files:
- '**/*'
name: CodeSource
Notice how the secondary artifact heading matches the output_artifacts. By ensuring that those heading match with the headings you ensure that the files passed are outputted correctly.