Gigaspaces LRMI Java Heap Space OutOfMemoryError

Question

I am getting a java.lang.OutOfMemoryError: Java heap space on light remote method invocation protocol (and watchdog). You can see the log below. Does anyone know why this would happen and how to solve it?

What I tried is changing the maximum heap size for the agent, the container, the lookup service and manager:

export GSA_JAVA_OPTIONS=-Xmx256m  
export GSC_JAVA_OPTIONS=-Xmx8192m  
export GSM_JAVA_OPTIONS=-Xmx4096m  
export LUS_JAVA_OPTIONS=-Xmx4096m  

This didn't help either. It makes sense though - the heap is failing because of lrmi related issue, but I don't see a setting that can help me solve it.

2013-10-05 21:01:53,124 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  
    at java.util.HashMap.newKeyIterator(HashMap.java:886)  
    at java.util.HashMap$KeySet.iterator(HashMap.java:920)  
    at java.util.HashSet.iterator(HashSet.java:170)  
    at sun.nio.ch.Util$2.iterator(Util.java:303)  
    at com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(AbstractSelectorThread.java:67)  
    at com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(AbstractSelectorThread.java:51)  
    at java.lang.Thread.run(Thread.java:722)  

2013-10-05 21:02:20,232 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:02:40,582 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:09,455 gs-space-0.0.2.2 [1] INFO [com.sun.jini.reggie] - unmarshalling failure; Caused by: java.lang.OutOfMemoryError: Java heap space  
2013-10-05 21:04:45,675 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:49,640 GSC SEVERE [com.gigaspaces.lrmi.watchdog] - Unexpected exception in watchdog thread.; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:05:05,617 gs-space-0.0.2.2 [1] WARNING [com.sun.jini.thread.TaskManager] - GS-JoinManager Task ran [net.jini.lookup.JoinManager$ProxyRegTask@69f9aca2] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space  
2013-10-05 21:04:47,283 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:40,335 GSC SEVERE [com.gigaspaces.lrmi] - exception in main selection loop; Caused by: java.lang.OutOfMemoryError: Java heap space  

2013-10-05 21:04:51,065  WARNING [com.sun.jini.thread.TaskManager] - GS-LookupDiscovery Task ran [net.jini.discovery.LookupDiscovery$DecodeAnnouncementTask@5164ce69] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space  
2013-10-05 21:05:21,328 gs-space-0.0.2.2 [1] WARNING [com.sun.jini.thread.TaskManager] - GS-JoinManager Task ran [net.jini.lookup.JoinManager$ProxyRegTask@4a8f3a1d] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space  
2013-10-05 21:05:14,649 gs-space-0.0.2.2 [1] WARNING [com.sun.jini.thread.TaskManager] - GS-JoinManager Task ran [net.jini.lookup.JoinManager$ProxyRegTask@3787ca39] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space  
2013-10-05 21:04:48,512 GSC WARNING [com.sun.jini.thread.TaskManager] - GS-LeaseRenewalManager Task ran [net.jini.lease.LeaseRenewalManager$RenewTask@4e833ee2] and caught an exception.; Caused by: java.lang.OutOfMemoryError: Java heap space  
2013-10-05 21:05:51,389 gs-space-0.0.2.2 [1] SEVERE [com.sun.jini.thread.TaskManager] - Ran [net.jini.lookup.JoinManager$ProxyRegTask@7c0469d7] but caught exception:   java.lang.OutOfMemoryError: Java heap space  
Attempt to log caused: ; Caused by: java.lang.OutOfMemoryError: Java heap space  

I have a 8 server configuration with 4 GSM's 4 LUS's 16 GSC's (or 8 GSC's) 8 primary partitions and 8 backups (or just 8 primaries). I am getting the error when I try to insert/write objects in the space one by one. When I reach around 200000 (the first partition that reaches this number, fails with the exception above. And I can see that the RAM is not yet field - there is still space in there (8GB - 3.5GB used - around 4GB free).

I also see some additional information in the GSC console (which I couldn't find in the log) about FastConcurrentSkipListMap's iterator causing the issue:

If anyone likes to visit the question on gigaspaces forum, its here.

Answer 1

The problem was that the jvm on the gsc-s was never made bigger because I used the "export" (linux) command for the .bat instead of the "set" (windows) command for environment variables. Once you make the jvm heap space bigger, this issue will go away.