I have an application with websocket and stomp as the protocol for messaging. I followed the official spring documentation in order to create this application. After a couple of days, i got a report that the host in which the application is running received an alert from Prometheus called FdExhaustionClose, which from my understanding means that some connections are not being properly closed.
The application is running in kubernetes (linux) and we are using RabbitMQ as the message broker.
How can i fix this ? Running locally i realized that the total number of connections since the application started actually matches the number of file descriptors unclosed.
I checked the logs in production and the broker status was printing this:
WebSocketSession[42 current WS(42)-HttpStream(0)-HttpPoll(0), 52766 total, 0 closed abnormally
And a simply lsof -p PID | wc -l returned the number 52752
WebSocketConfig
//...
@Configuration
public class WebSocketChatConfig extends DelegatingWebSocketMessageBrokerConfiguration {
@Autowired
private ApplicationProperties properties;
@Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/wss").setAllowedOrigins("*");
}
@Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
RabbitMqProperties rabbitProperties = properties.getRabbitmq();
ReactorNettyTcpClient<byte[]> client = new ReactorNettyTcpClient<>(tcpClient -> tcpClient
.host(rabbitProperties.getHost())
.port(rabbitProperties.getPort())
.option(ChannelOption.SO_TIMEOUT, 3600000)
.noProxy()
.secure(SslProvider.defaultClientProvider()), new StompReactorNettyCodec());
registry.setApplicationDestinationPrefixes("/app", "/topic", "/chat");
registry.enableStompBrokerRelay("/topic")
.setRelayHost(rabbitProperties.getHost())
.setRelayPort(rabbitProperties.getPort())
.setVirtualHost(rabbitProperties.getUsername())
.setSystemLogin(rabbitProperties.getUsername())
.setSystemPasscode(rabbitProperties.getPassword())
.setClientLogin(rabbitProperties.getUsername())
.setClientPasscode(rabbitProperties.getPassword()).setTcpClient(client);
}
}```
After some research i found out that the problem was related to the package reactor-netty, apparently there was a leak in version 0.9.8, the leak got fixed in version 0.9.9