How to properly set up Thread pool and maxMessagePerPoll in Spring Integration TaskExecutor for File adapter to avoid heap issue

Question

There's 2 issues I keep encountering and don't know how to resolve it.

1. GC overhead limit exceeded (Out of heap space, application currently using 2gb)
2. App pauses ThreadPoolExecutor Pool size= 50, active threads = 50, queued task= 50

From my understanding you need 1 thread per message poll. I set taskExecutor pool size to 50 and file adapter to fixed rate of 25 max message per poll. So, how does issue 2) happen if at most it'll use 25 threads out of 50?

I'm also using transaction-manager/sync factory to delete each file after processing. Does this have any impact on the above?

Answer 1

No, the logic there is like this:

            this.taskExecutor.execute(() -> {
                int count = 0;
                while (this.initialized && (this.maxMessagesPerPoll <= 0 || count < this.maxMessagesPerPoll)) {
                    if (this.maxMessagesPerPoll == 0) {
                        logger.info("Polling disabled while 'maxMessagesPerPoll == 0'");
                        break;
                    }
                    if (pollForMessage() == null) {
                        break;
                    }
                    count++;
                }
            });

So, one thread is used for the whole maxMessagesPerPoll batch. Apparently you have some how too many files to pull from directory. Plus it sounds like the processing for those files is too slow to pile up tasks in that ThreadPoolExecutor.

We need to see more of your configuration and flow logic to determine where exactly those threads are stuck. You can use JConsole to observer those memory leaks and blocked threads.