How to pass S3 continuation token from API Gateway to Lambda

Question

I have a REST API in AWS API Gateway that fetches json files on S3 and returns them to API Gateway via a Lambda function. I've implemented pagination in Lambda and I only fetch 100 files at a time, and returns the S3 Continuation Token if it exists.

I've created a query parameter on API Gateway to send the continuation token to S3 for the subsequent calls, but API Gateway won't deliver it because if its format.

A valid S3 token: 17kqE73kRqN1CZ8LJHWui7rlHD7+SHZgFRVQ5gVJAyF90msNxKgc6LiqRgMzGg5HfIJZP/Dc0Zenf8q7Az93KYtfkcRKtwQ6RL8P4cjGyjR0=

1. When using it directly in Postman, I get an "InvalidQueryStringException" error (http 400)
2. I found by testing directly in Lambda that removing the last '=' character gives the same file in S3 and no 400 error are thrown but the '+' character is replaced with a space in APIG which makes the token invalid.
3. I've also tried wrapping the token with "" and '' in Postman, without luck.
4. Calling the method directly in APIG via the test function providing the full token works as expected.

I map the continuation token from the url to the Lambda like this: "continuationToken": "$input.params('continuationToken')",

Any suggestions on how to get the token correctly passed API Gateway?

Answer 1

You need to urlencode the token when passing it as a query string, otherwise = will be treated as a parameter separator (subcomponent delimiter) as specified in RFC3986.

A properly encoded token would render the escaped = as %3D

If the Lambda was written in JS, you could encode the token like this:

const continuationToken = encodeURIComponent("17kqE73kRqN1CZ8LJHWui7rlHD7+SHZgFRVQ5gVJAyF90msNxKgc6LiqRgMzGg5HfIJZP/Dc0Zenf8q7Az93KYtfkcRKtwQ6RL8P4cjGyjR0=")
// continuationToken is now 17kqE73kRqN1CZ8LJHWui7rlHD7%2BSHZgFRVQ5gVJAyF90msNxKgc6LiqRgMzGg5HfIJZP%2FDc0Zenf8q7Az93KYtfkcRKtwQ6RL8P4cjGyjR0%3D