Search code examples
amazon-web-servicesamazon-s3aws-glueamazon-kinesis-firehose

Delivering Files from AWS Kinesis Firehose to a Specific S3 Folder

Problem Context:
I have an AWS Kinesis Firehose delivery stream setup.
I also have an S3 bucket setup.
Within my S3 bucket I have a folder, lets call it: "folderX/".

Currently, it just sends it to a premade folder labeled "2021/" because the default scheme is to send to a folder corresponding to the current date/time.

Question: I want my Firehose delivery stream to send files into that specific folder. How do I do that?

Solution

  • Backup S3 bucket prefix in S3 Destination configuration is the spot to set a prefix.

    From the Documentation for Backup S3 bucket prefix:

    To use the default prefix for Amazon S3 objects, leave this option blank. Kinesis Data Firehose automatically uses a prefix in "YYYY/MM/dd/HH" UTC time format for delivered Amazon S3 objects. You can add to the start of this prefix

    enter image description here

    Couple more examples:

    To get folderX/2021/3/17 we can use folderX/!{timestamp:YYYY}/!{timestamp:MM}/!{timestamp:dd}/

    To get folderX/year=2021/month=3/day=17/hour=5/ we can use folderX/year=!{timestamp:YYYY}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/