Search code examples
cachinghazelcast

Loading time of Hazelcast IMap


I have 1 Millions entries in the database and I have to put everything inside Hazelcast IMap.

So now how to test the loading time IMap?


Solution

  • There are a number of ways you can load the map; once you have the code that does the loading you can do something simple like calling System.currentTimeMillis() before and after the load to see how long it's taking.

    You can use the MapLoader functionality provided by Hazelcast to do loading on the member (cluster) side, or write your own JDBC code to load entries on the client side and add them to the cluster. You can also use Hazelcast Jet as a ingest/ETL engine, which is especially useful if you want to do filtering, enrichment, or data transformation of the data as it's being loaded.

    If you do loading from the client side, you really want to avoid the naive approach of iterating over the JDBC result set and putting entries to the map one at a time - doing a full round-trip to the cluster for each of a million entries will kill your performance. In a lab exercise from a training class I recently taught, I loaded an entire table of 55000 entries into a java.util.HashMap, and then did a IMap.putAll() to push the entire Map into Hazelcast in a single operation.

    Here's the code from the lab - it actually populates two tables with the same data (one has indexes, the other doesn't, which lets us compare performance in later labs that do queries and aggregations). It includes simple timing information on how long each load takes - not surprisingly, it's slower when we're building indexes for the tables, but then queries on the indexed table will be much faster.

    https://github.com/hazelcast/training-courses/blob/master/Special_Projects/mod02-db-etl-lab/src/main/java/com/hztraining/solutions/PopulateCacheWithJDBCSolution.java