apache-kafkaspring-kafka
Can't connect to Kafka broker using spring.kafka.boostrap-servers
I am starting a Kafka broker using docker compose.
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 22181:2181
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
As a sanity check, I connected to it using Offset Explorer.
However, it doesn't work in Spring Boot when specifying spring.kafka.boostrap-servers=localhost:29092 in application.properties.
2023-03-24T21:45:31.129-04:00 INFO 15008 --- [ main] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [localhost:9092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2023-03-24T21:45:31.499-04:00 INFO 15008 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version: 3.3.2
2023-03-24T21:45:31.502-04:00 INFO 15008 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: b66af662e61082cb
2023-03-24T21:45:31.503-04:00 INFO 15008 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1679708731495
2023-03-24T21:45:31.566-04:00 INFO 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2023-03-24T21:45:31.575-04:00 WARN 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2023-03-24T21:45:31.698-04:00 INFO 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2023-03-24T21:45:31.698-04:00 WARN 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2023-03-24T21:45:31.917-04:00 INFO 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2023-03-24T21:45:31.918-04:00 WARN 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2023-03-24T21:45:32.243-04:00 INFO 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Node -1 disconnected.
2023-03-24T21:45:32.243-04:00 WARN 15008 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
I had to specify that the zookeeper port is 22181 instead of 2181 to establish the connection, but I am not sure if it is important for Spring Boot since in the documentation they mainly specify props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");.
Edit: To keep it simple, the Spring Boot application wasn't running in a container.
Solution
It was simply a typo. Because of the typo, the default port was used bootstrap.servers = [localhost:9092]. I changed it to spring.kafka.bootstrap-servers=localhost:29092 and it works now.
2023-03-25T16:30:03.911-04:00 INFO 11840 --- [ main] o.a.k.clients.admin.AdminClientConfig : 2023-03-25T16:30:03.911-04:00 INFO 11840 --- [ main] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [localhost:29092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2023-03-25T16:30:04.615-04:00 INFO 11840 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version: 3.3.2
2023-03-25T16:30:04.619-04:00 INFO 11840 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: b66af662e61082cb
2023-03-25T16:30:04.621-04:00 INFO 11840 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1679776204610
2023-03-25T16:30:09.220-04:00 INFO 11840 --- [| adminclient-1] o.a.kafka.common.utils.AppInfoParser : App info kafka.admin.client for adminclient-1 unregistered
2023-03-25T16:30:09.236-04:00 INFO 11840 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics : Metrics scheduler closed
2023-03-25T16:30:09.236-04:00 INFO 11840 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-03-25T16:30:09.238-04:00 INFO 11840 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics : Metrics reporters closed
2023-03-25T16:30:09.283-04:00 INFO 11840 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 5000 (http) with context path ''
2023-03-25T16:30:09.300-04:00 INFO 11840 --- [ main] c.m.scores.ScoresApplication : Started ScoresApplication in 11.365 seconds (process running for 12.247)values:
bootstrap.servers = [localhost:29092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2023-03-25T16:30:04.615-04:00 INFO 11840 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version: 3.3.2
2023-03-25T16:30:04.619-04:00 INFO 11840 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: b66af662e61082cb
2023-03-25T16:30:04.621-04:00 INFO 11840 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1679776204610
2023-03-25T16:30:09.220-04:00 INFO 11840 --- [| adminclient-1] o.a.kafka.common.utils.AppInfoParser : App info kafka.admin.client for adminclient-1 unregistered
2023-03-25T16:30:09.236-04:00 INFO 11840 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics : Metrics scheduler closed
2023-03-25T16:30:09.236-04:00 INFO 11840 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-03-25T16:30:09.238-04:00 INFO 11840 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics : Metrics reporters closed
2023-03-25T16:30:09.283-04:00 INFO 11840 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 5000 (http) with context path ''
2023-03-25T16:30:09.300-04:00 INFO 11840 --- [ main] c.m.scores.ScoresApplication : Started ScoresApplication in 11.365 seconds (process running for 12.247)
- Kafka does not start blank output
- Data Loss in Flink Job with Iceberg Sink After Restart: How to Ensure Consistent Writes?
- Is there a limit on consumers and consumer groups?
- setting the value serializer only works via the constructor with 3 params, the map configuration is not considered
- Kafka 3.3 with Kraft - Controller ID keeps changing every second
- Kafka auto assigns new topics/partitions to dead/fenced brokers
- publishEndpoint.Publish() Not Sending Messages with Kafka in MassTransit
- Inconsistent Results with KStream-KTable Join Under Load
- Is it possible to create kafka topics without additional service in docker-compose?
- Apache Flink and State Store from Kafka Data Source
- Kafka topic creation uses wrong configuration
- Kafka TestContainer advertised listeners configuration when using DinD
- Run Kafka cluster with provectuslabs UI
- spring kafka No type information in headers and no default type provided
- python. kafka raft connect to controller
- How to increase __consumer_offsets replication number on an already running cluster (Kraft kafka 3.4.0)
- kafka Brokers Leader Skewed
- KafkIO: unable to create new "old" versions of schema in schema registry, no error
- PySpark: How To Deserialise A Proto Payload From A Kafka Message With Variable Message Type
- How to view Kafka messages?
- How do I run debezium with Confluent platform?
- How do I access the fields within a VARIANT column while reading from Kafka using Spark?
- Kafka schema registry - Broker: Broker failed to validate record
- How to find kafka broker username and password
- Kafka Connect REST Interface "PUT /connectors/(string: name)/config" Return Error Code 500
- How do you address messages coming out of order in a message queue?
- Setting up Apache Kafka Cluster
- What's the difference between RabbitMQ and kafka?
- Unknown feature gate KafkaNodePools found in the configuration
- Kafka max.request.size VS compression.type