Search code examples
amazon-s3aws-lambdaamazon-kinesisamazon-kinesis-firehoseaws-msk

Does firehose with MSK source and lambda transfomer support dynamic partitioning?


As per documentation I do not see mentioned anywhere that MSK does not support dynamic partitioning. However when I am creating firehose stream from console, I do not see enable dynamic partitioning option anywhere. If I change input to other values than MSK, it appears.

If I setup s3 prefix to use partitionKeyFromLambda, it gives error that dynamic partitioning is not enabled? Dynamic partitioning is not enabled, therefore, the prefix can't include !{partitionKeyFromQuery:keyID} or !{partitionKeyFromLambda:keyID}.

Is dynamic partitioning supported for MSK source and transformer lambda?

console error image


Solution

  • Confirmed with AWS support team. This is not supported yet from AWS side.