This is a curiosity question, recently we faced out of memory issue with one of our microservices. This memory issue happens once in a month so basically application is running for 1 month without any issue but after that it is throwing out of memory. Our guess memory is increasing after running through daily files and storing them into DB which not getting cleared by GC somehow.
We tried to refactor the code wherein I suggested to use custom array list (standard size) or singly linked list to store them. This made me wonder whether we have any memory efficient dynamic list to store large volume of custom objects.
Note: the list just need to store the record and access them in FIFO order, once recorded into DB it will be cleared
Note 2: please share if you know any memory efficient ADT already available in Apache commons collection or anything as such.
As @knittl suggested I took a heapdump and looks like Apache camel is causing this memory issue. We are pulling files from SFTP daily and decrypt them before going through data processing and inserting into DB. It looks like Apache camel is not freeing up the memory from its jsch buffer. Can someone suggest what we can do here?
Our camel is simple, sharing a sample here from("sftpUrl"+dynamic_fileName+".pgp").routeId("id").process(isDuplicateFile).unmarshal().pgp(somekey,user,pwd).unmarshal().string().process(writeToDBProcessor).log("Done");