How do I add delayed jobs on kafka? As I understand it doesn't deal with per message but per topic. My jobs have varying schedule in which I would like them to be consumed. Say one will be in the next 4 hours, another would be i Dec. 1, etc.
Does kafka have native support for this or other 3rd party ways to achieve the same?
I'm thinking of using Redis for the delayed queue instead, and push the job to kafka once its schedule has arrived but if possible I'd like to use only one dependency.
There is no notion of jobs in Kafka. It is just a dumb high performance message queueing service. Depending on your requirements you may consider storing the jobs in a storage that supports indexing by job execution time like some RDBMS. Then in some process periodically extract the jobs with execution times in some small range [last_check_time, current_time+lookahead_interval] and put them into a Kafka topic for eventual processing.