Search code examples
apache-kafkalogbackapache-kafka-streamsslf4j

Tag log messages per KafkaStreams instance


In an app, I have 3 KafkaStreams instances, and I would like to have each instance (including all other class instances it uses) log everything using a distinct marker, so that a log setting like the following will print the marker in the log.

<encoder>
  <pattern>...,%marker,...</pattern>

With that, I would be able to distinguish which instance a log message came from. (I already log thread ids, so I can group log messages, but I'd like to use a string to be able to associate a log with the instance that wrote it)

Is something like this even possible? It doesn't actually have to be implemented using slf4j markers, but that's the one Slf4j mechanism that I found works, if I have control over the code doing the logging - which I do not have in the case of the Kafka libraries.

N.B. My slf4j implementation currently is logback.


Solution

  • It turns out Kafka Streams is already logging the client.id, which can be set per instance either directly, or indirectly via application.id. It does not extend to all code run by an instance (e.g. AbstractConfig, AppInfoParser, StateDirectory, RocksDB), but seems good enough for me.