We want to use Kafka for all the communication between our services.
We have many high frequency topics (many messages per second), for which the Kafka mechanism of polling by consumers is efficient and suitable.
However, we also have some low frequency topics (no or few messages per day). If such a low frequency message is produced, all consumers should consume it immediately. But it would be inefficient if all consumers are polling with a high frequency (e.g. each second), while no or few such messages are produced each day. Most efficient would be if for such a low frequency topic the consumers can register a callback (e.g. a REST endpoint) with Kafka, and Kafka calls that callback when a message is produced. I cannot find such a callback mechanism in the Kafka documentation, however. How can we best resolve this in Kafka?
Consumers always poll. Kafka never pushes.
You could look into solutions like KEDA in Kubernetes or Serverless platforms that have Kafka integration, where you can scale your "processors" to zero, but the consumer itself is managed and always listening/polling for events