How to classifier topics/partitions in Kafka?

Question

For example user can subscribe on specific categories of films.

When a new films apperas in Kafka I must to send information about that to consumers who subscribes on category of film.

How to classifier this? Using partigion or topics? Because categories can be more over 1000.

Answer 1

Simply create a topic user-subscriptions that holds at least the userID and the categoryID. For each subscription, make sure to create a different record even for the same user. For example,

{"subscriptionID": 1, "userID": 1, "categoryID": 100}
{"subscriptionID": 2, "userID": 1, "categoryID": 26}
...

---

Now make sure to partition this topic by categoryID so that the records for the same category are placed in the same partition.

Now once your application identifies a new film of a particular category, you just need to go through the records of the partition that holds all the subscriptions for that category and while consuming, notify the users for the new film using the userID.