the title sums it up pretty much. I'm having issues connecting to my Kafka container from my microservice container. Everything runs nice and smoothly when I run the application locally, but when I start it using docker-compose up -d that is when the issues arise. So far I've tried creating a docker network, thinking that doing so would enable to containers to communicate with one another. Secondly, I've tried adding the Kafka bootstrap server URL to the Docker container, and that also did not work.
Like the stack trace for the microservice docker container states, there is an issue with my bootstrap.server URLs setting, but I can't exactly put my finger on what the issue actually is. Looking for some guidance, or even just another set of eyes on this issue.
I'll attach the Kafka and microservice docker-compose file, application.properties file, and also the stack trace below in that order. I should also add that I tried using the IP address of the Kafka docker container.
Thanks, everyone.
Kafka docker-compose.yml
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
networks:
- tappedtechnologies-network
broker:
image: confluentinc/cp-kafka:latest
container_name: broker
ports:
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_INTERNAL://broker:29092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
microservice docker-compose.yml
version: '3'
services:
api:
build: .
ports:
- "8088:8088"
environment:
- SPRING_DATASOURCE_URL=jdbc:mysql://user-service_mysqldb_1:3306/userservice_db
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=password
env_file:
- src/main/resources/application-dev.properties
depends_on:
- mysqldb
mysqldb:
image: mysql:latest
restart: always
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=userservice_db
volumes:
- dbdata:/var/lib/mysql
volumes:
dbdata:
application-dev.properties
# Kafka Properties
spring.kafka.bootstrap-servers=broker:9092
spring.kafka.properties.security.protocol=PLAINTEXT
spring.kafka.topic.users.saved=tappedtechnologies.users.saved
spring.kafka.topic.users.deleted=tappedtechnologies.users.deleted
# Kafka Consumer Properties
spring.kafka.consumer.group-id=userId
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.properties.spring.json.trusted.packages=*
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.deserializer.value.delegate.class=org.springframework.kafka.support.serializer.JsonDeserializer
# Kafka Producer Properties
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
spring.kafka.producer.properties.spring.json.type.mapping=saved:com.tappedtechnologies.userservice.events.UserSavedEvent, deleted:com.tappedtechnologies.userservice.events.UserDeletedEvent
Microservice Logs
Factory for persistence unit 'default'
2023-05-01T03:09:20.925Z INFO [Tapped Technologies - User Service,,] 1 --- [ main] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [broker:9092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2023-05-01T03:09:20.967Z WARN [Tapped Technologies - User Service,,] 1 --- [ main] org.apache.kafka.clients.ClientUtils : Couldn't resolve server broker:9092 from bootstrap.servers as DNS resolution failed for broker
2023-05-01T03:09:20.968Z ERROR [Tapped Technologies - User Service,,] 1 --- [ main] o.springframework.kafka.core.KafkaAdmin : Could not create admin
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:553) ~[kafka-clients-3.3.2.jar!/:na]
at org.apache.kafka.clients.admin.Admin.create(Admin.java:144) ~[kafka-clients-3.3.2.jar!/:na]
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49) ~[kafka-clients-3.3.2.jar!/:na]
at org.springframework.kafka.core.KafkaAdmin.createAdmin(KafkaAdmin.java:321) ~[spring-kafka-3.0.4.jar!/:3.0.4]
at org.springframework.kafka.core.KafkaAdmin.initialize(KafkaAdmin.java:201) ~[spring-kafka-3.0.4.jar!/:3.0.4]
at org.springframework.kafka.core.KafkaAdmin.afterSingletonsInstantiated(KafkaAdmin.java:183) ~[spring-kafka-3.0.4.jar!/:3.0.4]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:972) ~[spring-beans-6.0.4.jar!/:6.0.4]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:915) ~[spring-context-6.0.4.jar!/:6.0.4]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584) ~[spring-context-6.0.4.jar!/:6.0.4]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.0.2.jar!/:3.0.2]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:730) ~[spring-boot-3.0.2.jar!/:3.0.2]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:432) ~[spring-boot-3.0.2.jar!/:3.0.2]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-3.0.2.jar!/:3.0.2]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1302) ~[spring-boot-3.0.2.jar!/:3.0.2]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1291) ~[spring-boot-3.0.2.jar!/:3.0.2]
at com.tappedtechnologies.userservice.UserServiceApplication.main(UserServiceApplication.java:10) ~[classes!/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) ~[app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[app.jar:na]
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89) ~[kafka-clients-3.3.2.jar!/:na]
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48) ~[kafka-clients-3.3.2.jar!/:na]
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:505) ~[kafka-clients-3.3.2.jar!/:na]
... 23 common frames omitted
2023-05-01T03:09:21.008Z INFO [Tapped Technologies - User Service,,] 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8088 (http) with context path '/tapped-technologies/api/v0.1-dev'
2023-05-01T03:09:21.031Z INFO [Tapped Technologies - User Service,,] 1 --- [ main] c.t.userservice.UserServiceApplication : Started UserServiceApplication in 4.371 seconds (process running for 5.517)
DNS resolution failed for broker
Put all your services in one Docker compose file so that service name DNS will work. Otherwise, you must manually create a new Docker network and attach services to it.