Search code examples
amazon-web-servicesamazon-kinesis

Is it possible for 2 different partition keys to go into the same shard on kinesis?


I have a node app that puts events into a kinesis stream. If the partition key is different, is it possible for those 2 events to go into the same shard? The kinesis is configured to be on-demand


Solution

  • Yes it's possible, as you won't have a 1:1 mapping of shards and partition keys.

    Each shard will be responsible for a range of values. Your Partition key will MD5 hashed and that will determine which shard it belongs to.

    Partition Key

    A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings, with a maximum length limit of 256 characters for each key. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards using the hash key ranges of the shards. When an application puts data into a stream, it must specify a partition key.

    src