When first create an API Gateway deployment with the stage name, and also create a stage to configure X-RAY or CloudWatch logging, it will cause the "Stage already exist".
resource "aws_api_gateway_deployment" "this" {
rest_api_id = aws_api_gateway_rest_api.mysfit.id
stage_name = "${var.ENV}"
variables = {
deployed_at = timestamp()
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_api_gateway_stage" "this" {
stage_name = var.ENV
rest_api_id = aws_api_gateway_rest_api.mysfit.id
deployment_id = aws_api_gateway_deployment.this.id
dynamic "access_log_settings" {
for_each = var.enable_apigw_stage_cloudwatch_access_log ? [1] : []
content {
destination_arn = module.cloudwatch.cloudwatch_loggroup_arn
format = file("${path.module}/apigw_access_log_format.json")
}
}
xray_tracing_enabled = var.xray_tracing_enabled
tags = {
Project = var.PROJECT
Environment = var.ENV
}
}
The workaround is to omit the stage_name in aws_api_gateway_deployment as stage is optional for the API Gateway deployment. However, the the invoke_url of the deployment does not have the stage part in the URL path.
Is this a Terraform specific issue or an API Gateway issue?
In my understanding, there is a logical defect in the API Gateway deployment design.
API deployment is to deploy to a stage as in Deploying a REST API in Amazon API Gateway.
To deploy an API, you create an API deployment and associate it with a stage. A stage is a logical reference to a lifecycle state of your API (for example, dev, prod, beta, v2).
However, create-stage API requires an existing deployment. This circular dependency between deployment and stage is, I believe, the source of the problems.
The create-stage API requires a deployment with --deployment-id argument. Hence we need to create an API deployment first.
Here is the first problem. If we specifies a stage for create-deployment, it creates the stage. Then we cannot create the stage by ourselves.
When we use a configuration management tool e.g. CloudFormation or Terraform, this causes "Stage already exists" exception as we will try to create the stage resource ourselves.
Hence we cannot specify a stage when we first create a API deployment.
For us to manage the creation of the stage resource due to the problem 1, we need to first create a dummy deployment so that we can create a stage using the dummy. This step of wasteful deployment creation is the 2nd problem. Although the stage points to the deployment, the deployment does not fully recognize the stage because if try to get the invoke URL from the deployment, it does not include the stage.
Once the stage is created, then finally we can create another API deployment which specifies the stage. As the stage already exists, the deployment will refer to the stage and the invoke URL will include the stage.
#--------------------------------------------------------------------------------
# Dummy API Deployment
#--------------------------------------------------------------------------------
resource "aws_api_gateway_deployment" "dummy" {
rest_api_id = "${aws_api_gateway_rest_api.this.id}"
#--------------------------------------------------------------------------------
# To avoid State already exists
# https://github.com/terraform-providers/terraform-provider-aws/issues/2918
#--------------------------------------------------------------------------------
#stage_name = "${var.ENV}"
#--------------------------------------------------------------------------------
# Force re-deployment at each run. Alternative is to verify MD5 of API GW files.
#--------------------------------------------------------------------------------
# https://medium.com/coryodaniel/til-forcing-terraform-to-deploy-a-aws-api-gateway-deployment-ed36a9f60c1a
# https://github.com/hashicorp/terraform/issues/6613
# Terraform’s aws_api_gateway_deployment won’t deploy subsequent releases in the event
# that something has changed in an integration, method, etc
#--------------------------------------------------------------------------------
stage_description = "Deployment at ${timestamp()}"
lifecycle {
create_before_destroy = true
}
depends_on = [
#--------------------------------------------------------------------------------
# [aws_api_gateway_account.this]
# To avoid the error: Updating API Gateway Stage failed:
# BadRequestException: CloudWatch Logs role ARN must be set in account settings to enable logging.
#--------------------------------------------------------------------------------
"aws_api_gateway_account.this",
#--------------------------------------------------------------------------------
# To avoid NotFoundException: Invalid Integration identifier specified
#--------------------------------------------------------------------------------
"aws_api_gateway_integration.ping_put",
]
#--------------------------------------------------------------------------------
}
#--------------------------------------------------------------------------------
# Create a stage refering to the dummy.
# The 2nd/true deployment will later refer to this stage
#--------------------------------------------------------------------------------
resource "aws_api_gateway_stage" "this" {
stage_name = var.ENV
rest_api_id = aws_api_gateway_rest_api.this.id
deployment_id = aws_api_gateway_deployment.dummy.id
xray_tracing_enabled = var.apigw_xray_tracing_enabled
tags = {
Project = var.PROJECT
Environment = var.ENV
}
depends_on = [
aws_api_gateway_deployment.dummy
]
}
#--------------------------------------------------------------------------------
# Legitimate API Deployment
#--------------------------------------------------------------------------------
resource "aws_api_gateway_deployment" "this" {
rest_api_id = aws_api_gateway_rest_api.this.id
stage_name = aws_api_gateway_stage.this.stage_name
lifecycle {
create_before_destroy = true
}
}
AWSTemplateFormatVersion: "2010-09-09"
Description: "My API Gateway and Lambda function"
Parameters:
apiGatewayStageName:
Type: "String"
Default: "devStage"
lambdaFunctionName:
Type: "String"
Default: "my-lambda-function"
Resources:
apiGateway:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "test-api"
Description: "My Test API"
apiGatewayRootMethod:
Type: "AWS::ApiGateway::Method"
Properties:
AuthorizationType: "NONE"
HttpMethod: "GET"
Integration:
IntegrationHttpMethod: "POST"
Type: "AWS_PROXY"
Uri: !Sub
- "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
- lambdaArn: !GetAtt "lambdaFunction.Arn"
ResourceId: !GetAtt "apiGateway.RootResourceId"
RestApiId: !Ref "apiGateway"
apiGatewayDeployment:
Type: "AWS::ApiGateway::Deployment"
DependsOn:
- "apiGatewayRootMethod"
Properties:
RestApiId: !Ref "apiGateway"
StageName: ""
apiGatewayStage:
Type: "AWS::ApiGateway::Stage"
Properties:
StageName: !Ref "apiGatewayStageName"
RestApiId: !Ref "apiGateway"
DeploymentId: !Ref "apiGatewayDeployment"
MethodSettings:
- ResourcePath: /
HttpMethod: "GET"
MetricsEnabled: 'true'
DataTraceEnabled: 'true'
lambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Code:
ZipFile: |
def handler(event,context):
return {
'body': 'Hello World from Lambda',
'headers': {
'Content-Type': 'text/plain'
},
'statusCode': 200
}
Description: "My function"
FunctionName: !Ref "lambdaFunctionName"
Handler: "index.handler"
MemorySize: 256
Role: !GetAtt "lambdaIAMRole.Arn"
Runtime: "python3.7"
Timeout: 30
lambdaApiGatewayInvoke:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !GetAtt "lambdaFunction.Arn"
Principal: "apigateway.amazonaws.com"
SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/GET/"
lambdaIAMRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Action:
- "sts:AssumeRole"
Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Policies:
- PolicyDocument:
Version: "2012-10-17"
Statement:
- Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Effect: "Allow"
Resource:
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${lambdaFunctionName}:*"
PolicyName: "lambda"
lambdaLogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
LogGroupName: !Sub "/aws/lambda/${lambdaFunctionName}"
RetentionInDays: 90
Outputs:
apiGatewayInvokeURL:
Value: !Sub "https://${apiGateway}.execute-api.${AWS::Region}.amazonaws.com/${apiGatewayStageName}"
lambdaArn:
Value: !GetAtt "lambdaFunction.Arn"