spring-rabbitmq find a memory leak, but i don't know the reason, can someone help me

Question

my application use rabbit mq, the versions are below

spring-amqp : 2.2.17.RELEASE
spring-rabbit : 2.2.17.RELEASE
spring-boot: 2.3.11.RELEASE

    @Primary
    @Bean(name = "firstConnectionFactory")
    public ConnectionFactory firstConnectionFactory() {
        CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
        connectionFactory.setHost(host);
        connectionFactory.setPort(port);
        connectionFactory.setUsername(username);
        connectionFactory.setPassword(password);
        connectionFactory.setPublisherConfirms(true); 
        return connectionFactory;
    }

    @Primary
    @Bean(name = "firstRabbitTemplate")
    @Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
    public RabbitTemplate firstRabbitTemplate() {
        return new RabbitTemplate(firstConnectionFactory());
    }

    //send a message
    private void prepareForString(String exchange, String queueName, JSONObject message) {
        try {
            RabbitTemplate template = rabbitConfig.firstRabbitTemplate();
            template.convertAndSend(exchange, queueName, JSON.toJSONStringWithDateFormat(message,   DateUtils.YYYYMMDDHHMMSS, SerializerFeature.WriteMapNullValue));
        } catch (Exception ex) {
            log.error(ex.getMessage());
        }
    }

after the application run about 1 year, i find the application run very slow, dump the heap, then i see

heap show

One instance of "org.springframework.amqp.rabbit.connection.PublisherCallbackChannelImpl" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader @ 0x6c001a6d0" occupies 405.62 MB (12.57%) bytes. The memory is accumulated in one instance of "org.springframework.amqp.rabbit.connection.PublisherCallbackChannelImpl" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader @ 0x6c001a6d0". Keywords org.springframework.amqp.rabbit.connection.PublisherCallbackChannelImpl org.springframework.boot.loader.LaunchedURLClassLoader @ 0x6c001a6d0

it looks like the PublisherCallbackChannelImpl.java has a listeners, and the listeners is very huge. the listeners is defined below

//PublisherCallbackChannelImpl.java line 59
private final ConcurrentMap<String, PublisherCallbackChannel.Listener> listeners = new ConcurrentHashMap(); 

the below is the code of put elements in map

    //PublisherCallbackChannelImpl.java line 549
    public void addListener(PublisherCallbackChannel.Listener listener) {
        Assert.notNull(listener, "Listener cannot be null");
        if (this.listeners.size() == 0) {
            this.delegate.addConfirmListener(this);
            this.delegate.addReturnListener(this);
        }
        //the key code, below 'listener' is RabbitTemplate object
        if (this.listeners.putIfAbsent(listener.getUUID(), listener) == null) {
            this.pendingConfirms.put(listener, new ConcurrentSkipListMap());
            if (this.logger.isDebugEnabled()) {
                this.logger.debug("Added listener " + listener);
            }
        }

    }

i want know why this happens, and where is the listeners.remove(XXX), or it has no listeners.remove()?

i try look for the PublisherCallbackChannelImpl.java , but there is no listeners.remove() anywhere

so i what know, is this the reason of memory leak?

Answer 1

the reason of leak is here, SCOPE_PROTOTYPE and RabbitTemplate template = rabbitConfig.firstRabbitTemplate(); too many rabbitTemplate objects, and these rabbit template is in the map, and not gc by jvm, because the channel is not close, the map is related to the channel. so the map not gc by jvm. i think the designer of spring-rabbit not consider too many values in that map. the designer might think only one or serval rabbitTemplate objects, so the map will not too big. but when rabbitTemplate use SCOPE_PROTOTYPE or new every time, that will generate too many rabbitTemplate in the map, that cause the memory leak. I had changed to the below code.

//remove the @Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
//default singleton
@Primary
@Bean(name = "firstRabbitTemplate")
public RabbitTemplate firstRabbitTemplate() {
    return new RabbitTemplate(firstConnectionFactory());
}

//I use spring autowired to import RabbitTemplate Object 
@Autowired
RabbitTemplate firstRabbitTemplate;

/**
 * send message;
 */
private void prepareForString(String exchange, String queueName, JSONObject message) {
    try {
       //use just one firstRabbitTemplate obejct
       firstRabbitTemplate.convertAndSend(exchange, queueName, JSON.toJSONStringWithDateFormat(message,
                    DateUtils.YYYYMMDDHHMMSS, SerializerFeature.WriteMapNullValue));
    } catch (Exception ex) {
        log.error(ex.getMessage());
    }

}