I am trying to integrate the live reloading of k8s secrets in my spring boot 3 application.
My setup is as follows:
I have a Java 21 application that is being build as a Docker image containing an executable jar. The docker image is deployed on a k8s cluster using Helm and a secret is mounted in my deployment.yaml as follows:
- name: LIVE_RELOAD_SECRET
valueFrom:
secretKeyRef:
name: {{ .Chart.Name }}-live-secrets
key: TEST_LIVERELOAD_SECRET
The secret is present when I deploy (if I exec into the pod and do echo $LIVE_RELOAD_SECRET I get the expected value).
So far so good.
Now I want to be able to read this secret in my application AND also change it at runtime. I tried multiple solutions:
@RestController
@RefreshScope
public class SecretLogger {
private final PropertyResolver propertyResolver;
private String liveReloadSecret;
private String springCloudPropery;
@Autowired
public SecretLogger(@Value("${live-reload-secret}") String liveReloadSecret, @Value("${some.other.property}") String springCloudPropery, PropertyResolver propertyResolver) {
this.liveReloadSecret = liveReloadSecret;
this.propertyResolver = propertyResolver;
this.springCloudPropery = springCloudPropery;
}
@GetMapping("/secret")
public String liveReload() {
return "Injected value: " + liveReloadSecret + "\n" +
"PropertyResolver: " + propertyResolver.getProperty("live-reload-secret") + "\n" +
"System.getEnv: " + System.getenv("LIVE_RELOAD_SECRET") + "\n" +
"Spring cloud property: " + springCloudPropery;
}
}
I want to be able to call /refresh on my actuator endpoint and see the values change (by calling the /secret endpoint afterwards). I also added some.other.property, a spring-cloud property as a check (we also use spring-cloud-config-server for env specific properties).
The spring-cloud-config property:
some.other.property=someTestValue
So when everything is up and running and I call the /secret endpoint I first get this output:
Injected value: OYALELE
PropertyResolver: OYALELE
System.getEnv: OYALELE
Spring cloud property: someTestValue
This is in line with what I expect. The k8s secret has the value OYALELE and the spring-cloud-config value is also someTestValue.
Next I change the value of the secret in k8s and also the value in spring-cloud-config and I do a post to the /refresh endpoint on my deployed application.
The result is that the spring-cloud-config property has changed value, but the k8s secret seems to be the same. The only explanation I can think of is that you can not "live reload" env vars in a java application. But then I don't know how this https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/#monitoring-configmaps-and-secrets should ever work for secrets?
And I am not even using the configwatcher yet (that would be the next step). I don't see any reason to try the config watcher, as all it would do is automate the calling of the /refresh endpoint for me on changes to the secret. But if it does not work when I do it manually it will probably also not work otherwise.
I think I am just missing some logic or building blocks to live reload secrets on a k8s deployed spring boot 3 app but I can't get my head around them...
Any help would be much appreciated!
With the help of this thread I managed to get all of it working.
I will explain my mistakes and link an example project that uses kubernetes secrets and configmaps as properties and is able to refresh them after calling the /actuator/refresh endpoint.
My first mistake was trying to use "env" values in my deployment in k8s. Spring has no way of "updating" these with a context refresh as far as I know. What I ended up using was mounted secrets. Meaning that you mount the secret as a file on your container. To get this working you need to use 'spring.config.import: "configtree:/mnt/secrets/"', telling spring to look under a specified folder for properties. Alternatively or in combination you could also use 'spring.config.import: "configtree:/mnt/secrets/,kubernetes:,optional:configserver:"' if you want to use multiple property sources.
The "kubernetes:" part is used if you want to use the kubernetes discoveryClient to get secrets and configMaps, but you need the right RBAC rights for this (see example project).
Once I got this right, everything else started working as well. My example project does not include spring-cloud-kubernetes-config-watcher but once the /actuator/refresh endpoint works, spring-cloud-kubernetes-config-watcher should be a straight forward implementation following the docs.
My full implementation also included a SecretProviderClass that synced secrets from AzureKeyVault so I opted for file based properties because the SecretProviderClass already provides this implementation.