java wso2 jvm heap-memory wso2-enterprise-integrator

WSO2 EI Docker container 6.4.0 - Facing java.lang.OutOfMemoryError: GC overhead limit exceeded in ETL process

There is ETL process running in WSO2 EI 6.4.0.

Flow: Task Scheduler --> Proxy Service --> Sequence

Task scheduler will trigger proxy service based on cron expression(eg. 0 45 0/2 ? * * * )
Proxy service will fetch Data from source by using bigquery connector runQuery Operation and invoke sequence
Inside Sequence, Iterate mediator placed to iterate each record and framing payload to invoke DSS which will push data to target system.

Below is the sample sequence code for reference:

<iterate expression="$body//rows" id="BigQID" sequential="true">
            <target>
                <sequence>
                    <property name="FORCE_ERROR_ON_SOAP_FAULT" scope="default" type="STRING" value="true"/>
                    <property expression="//f[1]/v/text()" name="name" scope="default" type="STRING"/>
                    <property expression="//f[2]/v/text()" name="email" scope="default" type="STRING"/>
                    <property expression="//f[3]/v/text()" name="phone" scope="default" type="STRING"/>
                    
                    <!--  Frame payload for DSS Call -->
                    <header name="Action" scope="default" value="urn:wso2_dss_insert_op"/>
                    <payloadFactory media-type="xml">
                        <format>
                            <Envelope xmlns="http://www.w3.org/2003/05/soap-envelope">
                                <Body>
                                    <wso2_dss_insert_op xmlns="http://ws.wso2.org/dataservice">
                                        <name>$1</name>
                                        <email>$2</email>
                                        <phone>$3</phone>
                                    </wso2_dss_insert_op>
                                </Body>
                            </Envelope>
                        </format>
                        <args>
                            <arg evaluator="xml" expression="get-property('name')"/>
                            <arg evaluator="xml" expression="get-property('email')"/>
                            <arg evaluator="xml" expression="get-property('phone')"/>
                            
                        </args>
                    </payloadFactory>
                    <!--  calling DSS to push data to eMite CDW DB -->
                    <call>
                        <endpoint>
                            <http method="post" uri-template="http://localhost:8280/test-etl/services/WSO2_TEST_DSS/wso2_dss_insert_op">
                                <timeout>
                                    <duration>60000</duration>
                                </timeout>
                                <suspendOnFailure>
                                    <errorCodes>-1</errorCodes>
                                    <initialDuration>0</initialDuration>
                                    <progressionFactor>1.0</progressionFactor>
                                    <maximumDuration>0</maximumDuration>
                                </suspendOnFailure>
                                <markForSuspension>
                                    <retriesBeforeSuspension>0</retriesBeforeSuspension>
                                </markForSuspension>
                            </http>
                        </endpoint>
                    </call>
                </sequence>
            </target>
        </iterate>
        <property name="result" scope="default">
            <result xmlns=""/>
        </property>
        <!--  Aggregate for collecting DSS response in order to identify target updated row count -->
        <aggregate id="BigQID">
            <completeCondition>
                <messageCount max="-1" min="-1"/>
            </completeCondition>
            <onComplete aggregateElementType="root" enclosingElementProperty="result" expression="$body/*[1]">
                <log level="custom">
                    <property name="ON Aggregate SEQ" value="called***"/>
                    <property expression="fn:count(//*[local-name()='UpdatedRowCount'])" name="*********BigQueryETL-wso2_dss-TargetCount:: "/>
                </log>
            </onComplete>
        </aggregate>

DSS-Code:

 <data name="WSO2_TEST_DSS" transports="http https local">
  
   <config id="TestDataSource">
        <property name="carbon_datasource_name">WSO2_ETL_TEST_DB</property>
    </config>
 
   <query id="emp_insert_Q" returnUpdatedRowCount="true" useConfig="TestDataSource">
      <sql>insert into  empdata.employee (name,email,phone) values(?,?,?)</sql>
      <result element="UpdatedRowCount" rowName="" useColumnNumbers="true">
         <element column="1" name="Value" xsdType="integer"/>
      </result>
      <param name="name" sqlType="STRING"/>
      <param name="email" sqlType="STRING"/>
      <param name="phone" sqlType="STRING"/>
      
   </query>
   <operation name="wso2_dss_insert_op">
      <call-query href="emp_insert_Q">
         <with-param name="name" query-param="name"/>
         <with-param name="email" query-param="email"/>
         <with-param name="phone" query-param="phone"/>
      </call-query>
   </operation>
   
</data>

What I have understood is that when data volume exceeds more than 8000, Getting ERROR like java.lang.OutOfMemoryError: GC overhead limit exceeded or java.lang.OutOfMemoryError: Java heap space

ERROR Logs:

    [2023-07-18 13:43:17,828] [-1234] [] [PassThroughMessageProcessor-252] ERROR {org.apache.axis2.transport.base.threads.NativeWorkerPool} -  Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded 
at org.apache.synapse.commons.staxon.core.base.AbstractXMLStreamReader.readEndElementTag(AbstractXMLStreamReader.java:382)at org.apache.synapse.commons.staxon.core.json.JsonXMLStreamReader.consume(JsonXMLStreamReader.java:211)at org.apache.synapse.commons.staxon.core.json.JsonXMLStreamReader.consume(JsonXMLStreamReader.java:154)at org.apache.synapse.commons.staxon.core.json.JsonXMLStreamReader.consume(JsonXMLStreamReader.java:184)at org.apache.synapse.commons.staxon.core.base.AbstractXMLStreamReader.hasNext(AbstractXMLStreamReader.java:445)at org.apache.synapse.commons.staxon.core.base.AbstractXMLStreamReader.next(AbstractXMLStreamReader.java:455at org.apache.axiom.om.impl.builder.StAXOMBuilder.parserNext(StAXOMBuilder.java:681)at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:214)at org.apache.axiom.om.impl.llom.OMElementImpl.getNextOMSibling(OMElementImpl.java:336)at org.apache.axiom.om.impl.OMNavigator._getFirstChild(OMNavigator.java:199)at org.apache.axiom.om.impl.OMNavigator.updateNextNode(OMNavigator.java:140)at org.apache.axiom.om.impl.OMNavigator.getNext(OMNavigator.java:112)at org.apache.axiom.om.impl.SwitchingWrapper.updateNextNode(SwitchingWrapper.java:1113)at org.apache.axiom.om.impl.SwitchingWrapper.<init>(SwitchingWrapper.java:235)at org.apache.axiom.om.impl.OMStAXWrapper.<init>(OMStAXWrapper.java:74)at org.apache.axiom.om.impl.llom.OMStAXWrapper.<init>(OMStAXWrapper.java:52)at org.apache.axiom.om.impl.llom.OMContainerHelper.getXMLStreamReader(OMContainerHelper.java:51)at org.apache.axiom.om.impl.llom.OMElementImpl.getXMLStreamReader(OMElementImpl.java:736)at org.apache.axiom.om.impl.llom.OMElementImpl.cloneOMElement(OMElementImpl.java:1038)at org.apache.synapse.util.MessageHelper.cloneSOAPEnvelope(MessageHelper.java:690)at org.apache.synapse.mediators.eip.splitter.IterateMediator.mediate(IterateMediator.java:123)at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:108)at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:70)at org.apache.synapse.config.xml.AnonymousListMediator.mediate(AnonymousListMediator.java:37)at org.apache.synapse.mediators.filters.FilterMediator.mediate(FilterMediator.java:203)

at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:108)at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:70)at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)

at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:214)

To overcome issue, Clearing heap-dump.hprof in the <EI_HOME>/repository/logs file whenever encountered above ERROR and restarting the container
But intermittently facing similar issue again.
There is JVM configuration in Rancher where container management is done. JVM_Initial size is 8192 and JVM_Max size is 32768

Java version: 1.8.0_171

Is there any memory leak which we can control via code? or any way to resolve this issue in WSO2 EI?

Solution

In order to set the JVM memory params, one option is to add the following environment variable to the Pod, JVM_MEM_OPTS by setting the Xms and Xmx values.

JVM_MEM_OPTS="-Xms8192m -Xmx32768m"

Then on the server startup logs, you should see the following logs with the updated memory params.

CARBON_HOME environment variable is set to /home/xxx/xx/xxx/wso2ei-6.5.0
Using Java memory options: -Xms256m -Xmx1024m