spring kafka - set consumer log level manually (spring 5)

Question

We are working on spring project and integrated with spring-kafka recently. It's working as expected where we are able produce and consume messages properly.

But as we moved project to PROD, we are getting DEBUG level log statements and amount of statements overwhelmed. We are trying to set logging levels to WARN or ERROR specifically for Kafka logs.

In process we added logback.xml (added in main/resources) by following below article but no luck, also tried other solutions we came across in internet. (adding log4j.properties and log4j2.xml)

• https://www.baeldung.com/logback

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration name="default">

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
       <encoder>
          <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="org.apache.kafka"                 level="error" />
    <logger name="org.apache.kafka.common.metrics"  level="error"/>
    <logger name="org.apache.kafka.clients"         level="error"/>
    <logger name="org.apache.kafka.clients.consumer.internals"          level="error"/>
    <logger name="org.apache.kafka.clients.consumer.internals.Fetcher"  level="error"/>
     
    <root level="error" name="org.apache.kafka" additivity="false">
        <appender-ref ref="STDOUT" />
    </root> 
    
</configuration>

Sample logs:

[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.Metadata - [Consumer clientId=client-01, groupId=publisher] Cluster ID: Udhjf-fdbfds
[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer clientId=client-01, groupId=publisher] Discovered group coordinator 127.....0 (id: 65654654654 rack: null)
[org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator -  [Consumer clientId=client-01, groupId=publisher] (Re-)joining group
[WebSphere non-WLM Dispatch Thread t=008abcf0] WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.
[WebSphere non-WLM Dispatch Thread t=008abcf0] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.7.1
[WebSphere non-WLM Dispatch Thread t=008abcf0] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 616tyu85d0d567566

Anyone came across similar situation or any leads will be really helpful. Thanks in advance. :)

Answer 1

Late answer, but might be helpful for someone.

We did few changes in same logback.xml file and also added log4j2.xml in order to make it work in our application. Below are the changes from both files, please give it a try.

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration name="default">
   <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
     <encoder>
        <pattern> %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
     </encoder>
  </appender>
  
  <logger name="org.apache.log4j.xml"            level="error"/>
  <logger name="org.apache.http"                 level="error" />
  <logger name="org.apache.http.wire"            level="error"/>
  <logger name="org.apache.commons.httpclient"   level="error"/>
  
  <logger name="org.springframework" level="error">
      <appender-ref ref="STDOUT" />
  </logger> 
  
  <logger name="org.springframework.kafka" level="error" additivity="false">
      <appender-ref ref="STDOUT" />
  </logger>

  <logger name="org.apache.kafka" level="info" additivity="false">
      <appender-ref ref="STDOUT" />
  </logger>
   
  <root level="error">
      <appender-ref ref="STDOUT" />
  </root> 

</configuration>

log4j2.xml

<?xml version="1.0" encoding="UTF-8" ?>  
<!DOCTYPE log4j:configuration PUBLIC "-//log4j/log4j Configuration//EN" "log4j.dtd">  
<log4j:configuratio xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">  

   <appender name="consoleAppender" class="org.apache.log4j.ConsoleAppender">  
       <layout class="org.apache.log4j.PatternLayout">  
          <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" />  
       </layout>  
   </appender>  

   <appender name="fileAppender" class="org.apache.log4j.RollingFileAppender">  
       <param name="append" value="true" />  
       <param name="maxFileSize" value="10MB" />  
       <param name="maxBackupIndex" value="10" />  
       <param name="file" value="${catalina.home}/logs/myStruts1App.log" />  
       <layout class="org.apache.log4j.PatternLayout">  
          <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" />  
       </layout>  
   </appender>  

   <logger name="org.springframework">
      <level value="error" />
   </logger>

   <logger name="org.springframework.kafka">
      <level value="error" />
   </logger>

   <logger name="org.apache.http">
      <level value="warn" />
   </logger>
   
   <logger name="org.apache.http.wire">
      <level value="warn" />
   </logger>

   <logger name="org.apache.commons.httpclient">
      <level value="warn" />
   </logger>

   <logger name="org.apache.kafka">
      <level value="error" />
   </logger>

   <root>  
       <!-- DEBUG level for all other loggers -->
       <priority value="debug" />  
       <appender-ref ref="consoleAppender" />  
       <appender-ref ref="fileAppender" />  
   </root>  

</log4j:configuration>  

We can observe few changes here related to addivity, try to check below articles to know more about it.

• https://newsbasis.com/what-is-additivity-in-logback/

• Logback logger logging twice