Kafka starts throwing replication factor error as soon as spring boot app connects to it

Question

Kafka starts flooding logs with the the error

INFO [Admin Manager on Broker 2]: Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='compression.type', value='producer'), CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='segment.bytes', value='104857600')]) (kafka.server.ZkAdminManager) org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

As soon as the spring boot application connects. Here's the Kafka config I pass via application yaml

spring:
  cloud:
    stream:
      instanceIndex: 0
      kafka:
        default:
          producer:
            topic:
              replication-factor: 1
            configuration:
              key:
                serializer: org.apache.kafka.common.serialization.StringSerializer
              spring:
                json:
                  add:
                    type:
                      headers: 'false'
              value:
                serializer: org.springframework.kafka.support.serializer.JsonSerializer
              max:
                block:
                  ms: '100'
          consumer:
            topic:
              replication-factor: 1
            configuration:
              key:
                deserializer: org.apache.kafka.common.serialization.StringDeserializer
              spring:
                json:
                  trusted:
                    packages: com.message
              value:
                deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
        binder:
          autoCreateTopics: 'false'
          brokers: localhost:19092
          replicationFactor: 1
      bindings:
        consume-in-0:
          group: ${category}-${spring.application.name}-consumer-${runtime-env}
          destination: alm-tom-${runtime-env}
        publish-out-0:
          destination: ${category}-${spring.application.name}-${runtime-env}

I don't see any other config to control consumer-offsets topic replication factor

Answer 1

The error is coming from the broker itself. It will only start logging this when you create your first consumer group...

Check the server.properties file for offsets.topic.replication.factor.

Similarly, the transactions topic will have its own replication factor that needs reduced (assuming you're using a newer Kafka version)