amazon-web-services amazon-s3 aws-api-gateway

How do I handle 403/404 errors from S3 via API Gateway acting as an S3 proxy?

I have a Cloudformation template (see bottom) which allows me to fetch S3 objects directly via the API Gateway.

It works fine when getting objects that exist, returning a 200 and the object contents:

curl -i "https://xxx.execute-api.eu-west-1.amazonaws.com/prod/index.json"

HTTP/2 200 
content-type: application/json
...
{"hello": "world"}

However when the object doesn't exist, where I expect a 404 Not Found error, it still returns 200 OK:

curl -i "https:xxx.execute-api.eu-west-1.amazonaws.com/prod/abcd.json"

HTTP/2 200 
content-type: application/json
content-length: 243
...
<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
    <RequestId>CH2PRDQ2MVN0CXXX</RequestId>
    <HostId>cCV8dG+6CXA5pVrzTwLqiRqMjpuW8F+iuISQRUrKDP5PugEA6f4wauU0Egmb3b1GyvLjQZgjXXX=</HostId>
</Error>%

How do I modify my CloudFormation template so that a request for an unknown S3 key results in an HTTP 404 response?

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: S3ProxyAPI
  ApiGatewayResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: ApiGatewayRestApi
      ParentId:
        Fn::GetAtt:
          - ApiGatewayRestApi
          - RootResourceId
      PathPart: "{proxy+}"
  ApiGatewayMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      RestApiId:
        Ref: ApiGatewayRestApi
      ResourceId:
        Ref: ApiGatewayResource
      HttpMethod: GET
      RequestParameters:
        method.request.path.proxy: true
      MethodResponses:
        - StatusCode: 200
      Integration:
        IntegrationHttpMethod: ANY
        Type: AWS
        Uri:
          Fn::Sub: arn:aws:apigateway:${AWS::Region}:s3:path/${S3Bucket}/{proxy}
        Credentials:
          Fn::GetAtt:
            - ApiGatewayRole
            - Arn
        RequestParameters:
          integration.request.path.proxy: method.request.path.proxy
        PassthroughBehavior: WHEN_NO_MATCH
        IntegrationResponses:
          - StatusCode: 200
  ApiGatewayDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn: ApiGatewayMethod
    Properties:
      RestApiId:
        Ref: ApiGatewayRestApi
      StageName: prod
  ApiGatewayRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: ApiGatewayS3ProxyPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                Resource:
                  Fn::Sub: arn:aws:s3:::${S3Bucket}/*
Outputs:
  ApiEndpoint:
    Value:
      Fn::Sub: https://${ApiGatewayRestApi}.execute-api.${AWS::Region}.amazonaws.com/prod/{proxy}
  BucketName:
    Value:
      Ref: S3Bucket

Solution

MethodResponses define what HTTP status codes are possible for your API method. IntegrationResponses then handle the mapping of your 'backend' responses to API Gateway responses

They both work together.

You need a method response for the 404 status code to allow you to return the 404 status code & then an integration response with a selection pattern for 404. This way, you return a 404 response from your API Gateway for a 404 error from S3.

You're always getting 200 as there is no integration response to 'catch' the 404 error (or actually, 403 in this case but more on that below) and even if you did, you'd need a method response to be able to map it to xxx status code.

Modify your ApiGatewayMethod like so:

ApiGatewayMethod:
  Type: AWS::ApiGateway::Method
  Properties:
    MethodResponses:
      - StatusCode: 200
      - StatusCode: 404
    ...
    Integration:
      IntegrationResponses:
        - StatusCode: 200
        - StatusCode: 404
          SelectionPattern: '404'
      ...

However, this still won't work with the IAM policies you have defined - as per docs:

If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 ("access denied") error.

You don't have this IAM permission configured so AWS will always return 403 (as evident from <Code>AccessDenied</Code>) to prevent S3 directory enumeration. You'll never get a 404 for it to be mapped.

Ensure you also provide the s3:ListBucket permission like so:

Policies:
  - PolicyName: ApiGatewayS3ProxyPolicy
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Action:
            - s3:GetObject
          Resource:
            Fn::Sub: arn:aws:s3:::${S3Bucket}/*
        - Effect: Allow
          Action:
            - s3:ListBucket
          Resource:
            Fn::Sub: arn:aws:s3:::${S3Bucket}

This results in AWS actually returning a 404 error for files that are not found, and your API Gateway subsequently 'mapping' (or in this case, just passing through) the S3 404 response to a 404 method response.

The complete working YAML file is as below:

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: S3ProxyAPI
  ApiGatewayResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId:
        Ref: ApiGatewayRestApi
      ParentId:
        Fn::GetAtt:
          - ApiGatewayRestApi
          - RootResourceId
      PathPart: "{proxy+}"
  ApiGatewayMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      RestApiId:
        Ref: ApiGatewayRestApi
      ResourceId:
        Ref: ApiGatewayResource
      HttpMethod: GET
      RequestParameters:
        method.request.path.proxy: true
      MethodResponses:
        - StatusCode: 200
        - StatusCode: 404
      Integration:
        IntegrationHttpMethod: ANY
        Type: AWS
        Uri:
          Fn::Sub: arn:aws:apigateway:${AWS::Region}:s3:path/${S3Bucket}/{proxy}
        Credentials:
          Fn::GetAtt:
            - ApiGatewayRole
            - Arn
        RequestParameters:
          integration.request.path.proxy: method.request.path.proxy
        PassthroughBehavior: WHEN_NO_MATCH
        IntegrationResponses:
          - StatusCode: 200
          - StatusCode: 404
            SelectionPattern: '404'
  ApiGatewayDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn: ApiGatewayMethod
    Properties:
      RestApiId:
        Ref: ApiGatewayRestApi
      StageName: prod
  ApiGatewayRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: ApiGatewayS3ProxyPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                Resource:
                  Fn::Sub: arn:aws:s3:::${S3Bucket}/*
              - Effect: Allow
                Action:
                  - s3:ListBucket
                Resource:
                  Fn::Sub: arn:aws:s3:::${S3Bucket}
Outputs:
  ApiEndpoint:
    Value:
      Fn::Sub: https://${ApiGatewayRestApi}.execute-api.${AWS::Region}.amazonaws.com/prod/{proxy}
  BucketName:
    Value:
      Ref: S3Bucket

For successful responses, you'll get JSON and for failed responses, you'll get XML so you might need to tweak other parts of your CloudFormation so that it matches your Content-Type header (if needed).