Search code examples
amazon-web-servicesamazon-kinesis

Designing partition key to ensure data of particular type goes to same shard in Kinesis


I am new to AWS Kinesis data stream. In my use case, I would like to set up a Kinesis data stream with 2 shards, where one shard I would like to put the data related to say, Student, and in another shard, I would like to put the data related to Teacher.

Here, my question how can I design a partition key to ensure that data related to Student and Teacher don't land up in the same shard in the Kinesis data stream?


Solution

  • Each shard will be given fixed range of partition keys. You can get this range using describe-stream.

    For example, for the first shard:

                    "HashKeyRange": {
                        "StartingHashKey": "0",
                        "EndingHashKey": "113427455640312821154458202477256070484"
                    },
    

    Thus before you start streaming, you can programmatically check the ranges and choose key from the respective ranges.

    Having this you can use ExplicitHashKey when you PutRecords to explicitly specify which shard they should go.