How to assign specific data to user in Kafka?

Question

I have a some topic joined to Kafka Streams.

Stream analysis data and put result to another topic "Recommendations".

One recomendation can be assigned to on or more users.

• How to store recommendation in topic? By key in partition user_id?

• What if users are million?

• Could customer connects to stream avoiding topic?

Answer 1

How to store recommendation in topic? By key in partition user_id?

Depending on what you are trying to achieve, the key of the message should be the identifier that will be used to group messages in a single partition. Messages with the same key will be placed to the same partition.

---

What if users are million?

So your recommendations topic will have N number of partitions right? You decide to use user_id as the key of the message so all the recommendations of the same user are inserted to the same partition and this is absolutely fine.

Now I think what you are missing here is that you don't necessarily have to create a 1-1 mapping between users and partitions.