Search code examples
memoryramignite

Apache Ignite use too much RAM

I've tried to use Ignite to store events, but face a problem of too much RAM usage during inserting new data I'm runing ignite node with 1GB Heap and default configuration

curs.execute("""CREATE TABLE trololo (id LONG PRIMARY KEY, user_id LONG, event_type INT, timestamp TIMESTAMP) WITH "template=replicated" """);

n = 10000
for i in range(200):
    values = []
    for j in range(n):
    id_ = i * n + j
    event_type = random.randint(1, 5)
    user_id = random.randint(1000, 5000)
    timestamp = datetime.datetime.utcnow() - timedelta(hours=random.randint(1, 100))
    values.append("({id}, {user_id},  {event_type}, '{timestamp}')".format(
        id=id_, user_id=user_id, event_type=event_type, uid=uid, timestamp=timestamp.strftime('%Y-%m-%dT%H:%M:%S-00:00')
    ))
    query = "INSERT INTO trololo (id, user_id, event_type, TIMESTAMP) VALUES %s;" % ",".join(values)

    curs.execute(query)

But after loading about 10^6 events, I got 100% CPU usage because all heap are taken and GC trying to clean some space (unsuccessfully)

Then I stop for about 10 minutes and after that GC succesfully clean some space and I could continue loading new data

Then again heap fully loaded and all over again

It's really strange behaviour and I couldn't find a way how I could load 10^7 events without those problems

aproximately event should take:

8 + 8 + 4 + 10(timestamp size?) is about 30 bytes

30 bytes x3 (overhead) so it should be less than 100bytes per record

So 10^7 * 10^2 = 10^9 bytes = 1Gb

So it seems that 10^7 events should fit into 1Gb RAM, isn't it?

enter image description here

Solution

  • Actually, since version 2.0, Ignite stores all in offheap with default settings.

    The main problem here is that you generate a very big query string with 10000 inserts, that should be parsed and, of course, will be stored in heap. After decreasing this size for each query, you will get better results here.

    But also, as you can see in doc for capacity planning, Ignite adds around 200 bytes overhead for each entry. Additionally, add around 200-300MB per node for internal memory and reasonable amount of memory for JVM and GC to operate efficiently

    If you really want to use only 1gb heap you can try to tune GC, but I would recommend increasing heap size.