AWS Api Gateway to S3 integration how to list files under a folder

Question

I am working on building a simple REST API to serve files from S3 through API Gateway. I have followed the official aws guide and implemented the flow with AWS CDK. I am able to get a specific file from a nested folder, however, I am getting The specified key does not exist error when I call the path to list files in a specific folder.

My s3 bucket has a folder structure like; {top-level-folder}/{fileType}/{file}. And this API will consist of two GET methods. One will be for serving the list of files under {top-level-folder}/{fileType} and the other one is for getting the specific file from {top-level-folder}/{fileType}/{file}. As I said, getting the specific file is done, however, I couldn't achieved listing files.

const apiGateway = new apigw.RestApi(this, 'data-bucket-api', {
  restApiName: `data-bucket-rest-api`,
  description: 'REST API for serving raw data files',
  policy: new iam.PolicyDocument({
    statements: policyList,
  }),
  binaryMediaTypes: ['*/*'],
  minCompressionSize: cdk.Size.bytes(0),
});

// S3 integration to list objects within a folder with path of 'folder/fileType'
const s3IntegrationListObjects = new apigw.AwsIntegration({
  service: 's3',
  integrationHttpMethod: 'GET',
  path: `${dataBucket.bucketName}/{folder}/{fileType}`,
  options: {
    credentialsRole: executeRole,
    integrationResponses: [
      {
        statusCode: '200',
        responseParameters: {
          'method.response.header.Content-Type': 'integration.response.header.Content-Type',
        },
      },
    ],
    requestParameters: {
      'integration.request.path.folder': 'method.request.path.folder',
      'integration.request.path.fileType': 'method.request.path.fileType',
    },
  },
});

// path for 'folder/fileType'
const fileTypeResource = apiGateway.root.addResource('{folder}').addResource('{fileType}');

// Add GET method for list objects
fileTypeResource.addMethod('GET', s3IntegrationListObjects, {
  authorizationType: apigw.AuthorizationType.IAM,
  methodResponses: [
    {
      statusCode: '200',
      responseParameters: {
        'method.response.header.Content-Type': true,
      },
    },
  ],
  requestParameters: {
    'method.request.path.folder': true,
    'method.request.path.fileType': true,
    'method.request.header.Content-Type': true,
  },
});

// S3 integration for get objects with path folder/fileType/key
const s3Integration = new apigw.AwsIntegration({
  service: 's3',
  integrationHttpMethod: 'GET',
  path: `${dataBucket.bucketName}/{folder}/{fileType}/{key}`,
  options: {
    credentialsRole: executeRole,
    integrationResponses: [
      {
        statusCode: '200',
        responseParameters: {
          'method.response.header.Content-Type': 'integration.response.header.Content-Type',
        },
      },
    ],
    requestParameters: {
      'integration.request.path.folder': 'method.request.path.folder',
      'integration.request.path.fileType': 'method.request.path.fileType',
      'integration.request.path.key': 'method.request.path.key',
    },
  },
});

// path for 'folder/fileType/key'
const objectKeyResource = fileTypeResource.addResource('{key}');

// add GET method for getting objects
objectKeyResource
  .addMethod('GET', s3Integration, {
    authorizationType: apigw.AuthorizationType.IAM,
    methodResponses: [
      {
        statusCode: '200',
        responseParameters: {
          'method.response.header.Content-Type': true,
        },
      },
    ],
    requestParameters: {
      'method.request.path.folder': true,
      'method.request.path.fileType': true,
      'method.request.path.key': true,
      'method.request.header.Content-Type': true,
    },
  });

Thanks your help!

Answer 1

As @gshpychka pointed out listing objects can be done by using ListBucket action with a filtering support on the prefix parameter. I defined a query parameter called prefix and the related api calls S3 with ListBucket action.

    const s3IntegrationListObjects = new apigw.AwsIntegration({
      service: 's3',
      integrationHttpMethod: 'GET',
      subdomain: crossAccDataBucket.bucketName,
      action: 'ListBucket',
      options: {
        credentialsRole: executeRole,
        integrationResponses: [
          {
            statusCode: '200',
            responseParameters: {
              'method.response.header.Content-Type': 'integration.response.header.Content-Type',
            },
          },
        ],
        requestParameters: {
          'integration.request.querystring.prefix': 'method.request.querystring.prefix',
        },
      },
    });