We have a spring api gateway which routes the request to a kubernetes service running 3 pods. Requests coming to the gateway are being routed successfully. Now, the number of pods is increased to 5. Both the pods have been successfully registered (this can be checked in the kubernetes environment), but the new pods are not requesting any traffic.
When we refresh the api gateway from spring boot admin, all the pods start receiving traffic again.
Issue was fixed by the following code change. Apparently the ips of the pods are being cached on api gateway. From what I know, api gateway will send request to service and even if it caches the service ip, that should be static weather or not new pods are added. Would be helpful if someone could explain this to me. However, the code fix is here. And this is the github issue where I found this from.
@Component
public class DnsCacheCustomizer implements HttpClientCustomizer {
@Override
public HttpClient customize(HttpClient httpClient) {
DnsNameResolverBuilder dnsResolverBuilder = new DnsNameResolverBuilder()
.channelFactory(EpollDatagramChannel::new)
.resolveCache((NoopDnsCache.INSTANCE));
httpClient = httpClient.resolver(new DnsAddressResolverGroup(dnsResolverBuilder));
return httpClient;
}
}