I'm attempting to take advantage of Spring Cloud Kubernetes in my Spring Boot-based microservice, namely autoconfiguration and service discovery.
However, I get an error message (Timeout waiting for informers cache to be ready, is the kubernetes service up?) during the initialization, as well as high verbosity, both of which I am unable to find information online to help me pinpoint the cause. Error message and excessive log messages are detailed below. Apologies for the lack of context, but I'm also puzzled!
The application is running within the default service account, which has all the permissions specified in the documentation:
$ kubectl describe serviceaccount default
Name: default
Namespace: joaomlneto
Labels: <none>
Annotations: <none>
Image pull secrets: devspace-auth-rg-nl-ams-scw-cloud
Mountable secrets: default-token-2sxvc
Tokens: default-token-2sxvc
Events: <none>
$ kubectl describe rolebinding namespace-reader-binding
Name: namespace-reader-binding
Labels: <none>
Annotations: <none>
Role:
Kind: Role
Name: namespace-reader
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount default joaomlneto
$ kubectl describe role namespace-reader
Name: namespace-reader
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
configmaps [] [] [list watch get]
endpoints [] [] [list watch get]
pods [] [] [list watch get]
secrets [] [] [list watch get]
services [] [] [list watch get]
configmaps.apps [] [] [list watch get]
endpoints.apps [] [] [list watch get]
pods.apps [] [] [list watch get]
secrets.apps [] [] [list watch get]
services.apps [] [] [list watch get]
configmaps.extensions [] [] [list watch get]
endpoints.extensions [] [] [list watch get]
pods.extensions [] [] [list watch get]
secrets.extensions [] [] [list watch get]
services.extensions [] [] [list watch get]
I'm using Spring Boot 2.4; Spring Cloud Kubernetes 2.0.2. My pom.xml:
<properties>
<java.version>11</java.version>
<spring-cloud.version>2020.0.2</spring-cloud.version>
<spring-cloud-kubernetes.version>2.0.2</spring-cloud-kubernetes.version>
<graphql-java-kickstart.version>11.0.0</graphql-java-kickstart.version>
<lombok.version>1.18.20</lombok.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-client-all</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.graphql-java-kickstart</groupId>
<artifactId>graphql-spring-boot-starter</artifactId>
<version>${graphql-java-kickstart.version}</version>
</dependency>
<dependency>
<groupId>com.graphql-java-kickstart</groupId>
<artifactId>graphql-spring-boot-starter-test</artifactId>
<version>${graphql-java-kickstart.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-kubernetes-dependencies</artifactId>
<version>${spring-cloud-kubernetes.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludeDevtools>false</excludeDevtools>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>Maven Central</name>
<url>https://repo1.maven.org/maven2</url>
</repository>
<repository>
<id>supersecret</id>
<name>A Super Secret Repo</name>
<url>https://example.com/1</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<distributionManagement>
<repository>
<id>supersecret2</id>
<name>Another Super Secret Repo</name>
<url>https://example.com/2</url>
</repository>
</distributionManagement>
However, a minute after the application starts successfully, I get the following error:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kubernetesInformerDiscoveryClient' defined in class path resource [org/springframework/cloud/kubernetes/client/discovery/KubernetesDiscoveryClientAutoConfiguration$KubernetesInformerDiscoveryConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.5.jar:5.3.5]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.5.jar:5.3.5]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[spring-boot-2.4.4.jar:2.4.4]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:769) ~[spring-boot-2.4.4.jar:2.4.4]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) ~[spring-boot-2.4.4.jar:2.4.4]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) ~[spring-boot-2.4.4.jar:2.4.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) ~[spring-boot-2.4.4.jar:2.4.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1313) ~[spring-boot-2.4.4.jar:2.4.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1302) ~[spring-boot-2.4.4.jar:2.4.4]
at myorg.myapp.Application.main(Application.java:9) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.4.4.jar:2.4.4]
Caused by: java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?
at org.springframework.cloud.kubernetes.client.discovery.KubernetesInformerDiscoveryClient.afterPropertiesSet(KubernetesInformerDiscoveryClient.java:221) ~[spring-cloud-kubernetes-client-discovery-2.0.2.jar:2.0.2]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1845) ~[spring-beans-5.3.5.jar:5.3.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782) ~[spring-beans-5.3.5.jar:5.3.5]
... 22 common frames omitted
I also get the following messages logged every second (both before and after the error, ad infinitum):
2021-04-18 16:19:12.585 INFO 224 --- [pool-9-thread-1] .k.c.d.KubernetesInformerDiscoveryClient : Waiting for the cache of informers to be fully loaded..
2021-04-18 16:19:12.935 INFO 224 --- [s.V1Endpoints-1] i.k.c.informer.cache.ReflectorRunnable : class io.kubernetes.client.openapi.models.V1Endpoints#Start listing and watching...
2021-04-18 16:19:12.950 INFO 224 --- [els.V1Service-1] i.k.c.informer.cache.ReflectorRunnable : class io.kubernetes.client.openapi.models.V1Service#Start listing and watching...
I managed to get rid of the error messages by using the Fabric8 client instead of the official client. This simple change was enough to get rid of my problems.
This was done by changing the dependency in pom.xml from spring-cloud-starter-kubernetes-client-all to spring-cloud-starter-kubernetes-fabric8-all.
As this doesn't seem like expected behavior, I opened a bug report on the Spring Cloud Kubernetes project on GitHub.
EDIT Jan/2022: The bug report has been marked as solved, although some people still experience the issue. I am still using Fabric8 and haven't ran into any issues. Haven't looked back.