Search code examples
amazon-kinesisamazon-kinesis-firehose

Firehose - JSON attribute on S3 prefix


I'm setting up a Firehose that will delivery JSON messages to a bucket S3. The setup was pretty easy and it's working but I would like to add a prefix on S3 like <event>/YYYY/MM/DD/HH.

I'm reading the docs but i didn't found a way to get an attribute value from my json and use it on prefix. I would like to use a custom prefix to split my events type.

Is possible to do it?


Solution

  • Each delivery stream can be configured to have only a single prefix (https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#s3-object-name):

    "S3Configuration": { 
         "BucketARN": "string",
         "BufferingHints": { 
            "IntervalInSeconds": number,
            "SizeInMBs": number
         },
         "CloudWatchLoggingOptions": { 
            "Enabled": boolean,
            "LogGroupName": "string",
            "LogStreamName": "string"
         },
         "CompressionFormat": "string",
         "EncryptionConfiguration": { 
            "KMSEncryptionConfig": { 
               "AWSKMSKeyARN": "string"
            },
            "NoEncryptionConfig": "string"
         },
         "Prefix": "string",
         "RoleARN": "string"
      }
    

    A possible workaround can be to filter events in your application code and send them to a correct delivery stream.