Search code examples
memory-leaksgarbage-collectionjava-8titanapache-tomee

Titan 1.0.0 graph cache memory leaks

I am running TomEE (kind of Tomcat) server on Java 8 and using Titan 1.0.0 with Cassandra backend and Elasticsearch index as my database. In my development environment with small user connection everything works perfect but on my production server with many user connected with many db access we have an issue with memory leaks probably because Titan graph cache or something Titan relative. I have configure 4GB heap-size and after more than 10 hours of run allocated memory on heap grows to its maximum (~300Mb per hour) and it causes that GC (Garbage Collector) can't clean up anymore and remains to run continuously that causes server instance to be unresponsive. Using VisualVM i did some memory profiling and those are my screenshots: Allocated memory graph
Heap histogram Any suggest how we can fix this or find the way to investigate this issue in more details? May be some GC params can help us in this case?

Solution

  • I have seen those problems before on Titan 1.0 with Cassandra. Two things to check:

    Opening And Closing The Graph

    Are you opening different transaction to the graph per user or different graphs per user? i.e. are you doing:

    (1)

    //User 1 Logs in
graph = TitanFactory.open(config);
soStuffWithGraph(graph);
//User 2 Logs in
graph = TitanFactory.open(config);
soStuffWithGraph(graph);

    or

    (2)

    graph = TitanFactory.open(config);
//User 1 Logs in
soStuffWithGraph(graph);
//User 2 Logs in
soStuffWithGraph(graph);

    Approach (1) means that each user gets their own connection to the graph with their own graph object. This is very heavy and leads to more memory usage.

    Approach (2) means that every user uses the same connection but different transactions. This is preferable in my opinion. Note: This is assuming users are on different threads.

    Long lived Transactions

    This is the problem I had which resulted in similar GC problems to yourself. I simply kept transactions alive for too long. To speed up querying Titan caches a lot and I don't think it clears the cache unless the transaction is closed. So ideally you should have something like:

    graph = TitanFactory.open(config);
//User 1 Logs in
soStuffWithGraph(graph);
graph.tx().close();
//User 2 Logs in
soStuffWithGraph(graph);
graph.tx().close();

    where each transaction is closed after a user is done with it.