Spring Rabbit - Semaphore permit leak leads to "No available channels" exception

Question

We use CachingConnectionFactory for our consumers. With every connection drop I've seen that one checkoutPermit gets acquired and it's never released. So let's say that if we go with default cache channel size of 25, next time, when connection is recovered after the drop, number of available permits will be 24. After some time that leads to number of permits being 0 hence causing the exception AmqpTimeoutException("No available channels").

I've observed this behavior in versions 1.6.10-RELEASE, 1.7.3-RELEASE and 2.0.0-BUILD-SNAPSHOT.

Is it possible that we're using the library in the wrong way and we should take care of the manual releasing of checkoutPermit, possibly by closing channels on our own? (releasePermitIfNecessary is never being called after a connection drop)

Thanks in advance.

---

Example (Using 1.7.3-RELEASE)

Configuration

@Configuration
public class Config {

    @Bean
    public CachingConnectionFactory cachingConnectionFactory() {
        CachingConnectionFactory connectionFactory = new CachingConnectionFactory("localhost");
        connectionFactory.setUsername("username");
        connectionFactory.setPassword("password");
        connectionFactory.setVirtualHost("vhost");
        connectionFactory.setChannelCheckoutTimeout(1200);
        connectionFactory.setConnectionTimeout(1000);
        connectionFactory.setPort(5672);
        return connectionFactory;
    }

    @Bean
    public SimpleMessageListenerContainer simpleMessageListenerContainer(CachingConnectionFactory cachingConnectionFactory) {
        SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
        container.setConnectionFactory(cachingConnectionFactory);
        container.setQueueNames("test.queue");
        container.setMessageListener(new MessageListenerAdapter(new TestHandler()));
        container.start();
        return container;
    }

}

Handler (just for the sake of testing)

public class TestHandler {

    public String handleMessage(byte[] textBytes) {
        String text = new String(textBytes);
        System.out.println("Received: " + text);
        return text;
    }

}

I test the connection drop by using proxy between RabbitMQ and my app, where I manually break the connection to RabbitMQ.

Answer 1

Confirmed.

That's definitely bug. When we lose the connection we lose all the channels as well. Therefore we have to reset permits associated.

Please, raise a JIRA ticket with the proper description.

Meanwhile I guess as a workaround you should not use etChannelCheckoutTimeout(1200) and leave it as 0, what is default value:

/**
 * Sets the channel checkout timeout. When greater than 0, enables channel limiting
 * in that the {@link #channelCacheSize} becomes the total number of available channels per
 * connection rather than a simple cache size. Note that changing the {@link #channelCacheSize}
 * does not affect the limit on existing connection(s), invoke {@link #destroy()} to cause a
 * new connection to be created with the new limit.
 * <p>
 * Since 1.5.5, also applies to getting a connection when the cache mode is CONNECTION.
 * @param channelCheckoutTimeout the timeout in milliseconds; default 0 (channel limiting not enabled).
 * @since 1.4.2
 * @see #setConnectionLimit(int)
 */
public void setChannelCheckoutTimeout(long channelCheckoutTimeout) {