I have a cfc method that is looping through a list and making a series of SOAP calls via cfhttp. Then inserts the result into a database.
The process itself works great the problem is that the java memory slowly fills up and eventually (depending on the number of elements in the records being returned) just stops working. There is no error or anything visible it just stops. If I look at the application log file via the coldfusion admin I see one or both of the following errors:
GC overhead limit exceeded The specific sequence of files included or processed is:
or
Java heap space The specific sequence of files included or processed is:
Below is simplified version of the code I am running:
<cfsetting requesttimeout="3600">
<cfloop condition="thisPass lt 10">
<cfscript>
runtime = CreateObject("java","java.lang.Runtime").getRuntime();
objSystem = CreateObject( "java", "java.lang.System" );
soapBody = '';
soapResponse = '';
thisStruct = '';
lock scope='application' type='exclusive' timeout='60' {
//This is where I am trying to manage the memory and call garbage collection
try {
freeMemory = runtime.freeMemory()/1024/1024;
writeOutput("- fm = "&freeMemory);
if (freeMemory < 200){
objSystem.gc();
sleep(1000);
writeDump(' - dumping freeMemory');
}
}
catch(any error) {
writeDump(' - trying to dump GC as '&now()& ' freeMemory = '&freeMemory);
}
}
</cfscript>
<cfsavecontent variable="soapBody">
<?xml version="1.0" encoding="utf-8"?>
[ BUILD SOAP ENVELOP ]
</cfsavecontent>
<cfhttp url="[URL]" method="post" result="httpResponse"
timeout="600" resolveurl="false">
<cfhttpparam type="header" name="SOAPAction" value="[URL2]" />
<cfhttpparam type="xml" value="#trim( soapBody )#"/>
</cfhttp>
<cfscript>
soapBody = "";
soapResponse = httpResponse.fileContent;
soapResponse = xmlParse( soapResponse );
thisStruct = xmlSearch(soapResponse,'/soap:Envelope/soap:Body/')[1].xmlChildren[1].xmlChildren[1].xmlChildren;
writeOutput("-"&arrayLen(thisStruct)&' records');
getPageContext().getOut().flush();
if(arrayLen(thisStruct) == 2500){
thisPass = thisPass+1;
} else {
writeOutput("- total records = "&(2500*(thisPass-1))+arrayLen(thisStruct));
thisPass = 100; // since looping while thisPass lt 10 this should prevent the next iteration
}
</cfscript>
<cfloop from="1" to="#arrayLen(thisStruct)#" index="i">
[RUN PROC TO INSERT RECORDS]
</cfloop>
</cfloop>
The GC seems to sometimes release a bit of memory but not with any reliability. I understand that GC() is only a recommendation for java to release some of the unused memory but I am unsure how I can get it to FORCE it to release memory. It is possible that there is a leak somewhere but I am not seeing it. I am hoping that this is something obvious that I have over looked and I admit that my java knowledge is extremely limited.
Is there some java guru out there that can see my error?
there are 236 lists to loop through
After a lot searching I found that the best "fix" for this particular problems was to remove the code that was attempting to do the garbage collection and increase the java heap sizes. In the /jrun/bin/jvm.config file.
By changes the Arguments to VM to :
java.args=-server -Xms2048m -Xmx2048m -Xmn1024m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/ -Djava.security.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/coldfusion.policy -Djava.security.auth.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/neo_jaas.policy
I was able to increase the initial heap size (Xms) and the max heap size (Xmx) to 2048m and the and the "young generation" heap size (Xmn) to 1024m Since it was suggested to me that the young generation should be smaller than the initial and max for better garbage collection.
As suggested by James, I commented out the actual processes (which are in a function and var'd) and then uncommenting them one at a time running everything each time. What I learned by this was that the large SOAP responses were what was filling up the memory and not a leak as I had feared.
As Adam mentioned, it it wasn't about Java managing the GC but rather there wasn't enough room to manage was I was throwing at at (for some reason CF doesn't like to deal with 2500 record SOAP responses very well) go figure.
Adam was also correct saying that trouble shooting Java memory in CF is a "dark art" By using the server monitor http://localhost/CFIDE/administrator/monitor/launch-monitor.cfm and going to the statistics tab under Memory Usage -> Memory Usage Summary I could watch the memory slowly fill up and dump itself even when after a fresh reboot no processes where running. Never was able to figure out why but I could see what levels were running and that I was hitting the top.
The default memory allocated in the jvm.config file was 512m and simply wasn't enough to handle what was going on. There may be a better way to handle this overall process. May have to implement Henry's suggestion and put it in the database and chunk through it, although that sounds very clumsy to me.
I am sure that other then CF now taking up a huge chunk of resources right out of the gate there may be other problems with this solution but (for now) it seems to be working as needed.
UPDATE : I changed the cfc function so that instead of inserting everything into the database I wrote all the XML to files and and then read the file and insert into the database. Somehow the writing to file allowed java to "breath" long enough to do the GC() itself.