Hazelcast IMDG vs Hazelcast Jet config

Question

How to read Hazlecast IMDG data in the Hazelcast jet.

In my case I required both Hazlecast IMDG (Distributed cache) to store data for the future and also jet to perform batch and stream processing.

So I will be saving data using Hazelcast IMDG(MapStore) and filtering using Hazelcast jet.

public class Test {

    HazelcastInstance hz = Hazelcast.newHazelcastInstance();
    JetInstance jet = Jet.newJetInstance();

    public static void main(String[] args) {
        Test t = new Test();
        t.loadIntoIMap();
        t.readFromIMap();
    }

    public void loadIntoIMap() {
        IMap<String, String> map = hz.getMap("my-distributed-map");
        // Standard Put and Get
        map.put("1", "John");
        map.put("2", "Mary");
        map.put("3", "Jane");
    }

    public void readFromIMap() {
        System.err.println("--manu---");
        jet.getMap("s").put("1", "2");
        System.err.println(jet.getMap("s").size());
        System.err.println(jet.getMap("my-distributed-map").size());
    }

}

Do we need separate configuration both(jet and IMDG) or in single config can I share Hz IMap data inside jet.

I'm little confused between jet and Hazelcast IMDG

Answer 1

The answer differs depending on the version you want to use.

IMDG up to 4.2 and Jet 4.5

Hazelcast Jet is built on top of Hazelcast IMDG. When you start a Jet instance there is automatically an IMDG instance running. There is JetInstance#getHazelcastInstance method to retrieve the IMDG instance from Jet instance and JetConfig#setHazelcastConfig to configure IMDG specific configs.

You can access the maps from your cluster in Jet using com.hazelcast.jet.pipeline.Sources#map(String)

You should not start both IMDG and Jet separately on the same machine. However, you can create 2 clusters, one IMDG, one Jet and connect from Jet using com.hazelcast.jet.pipeline.Sources#remoteMap(String, ClientConfig) and similar for other data structures.

If you are already using Hazelcast it's likely this version.

Hazelcast 5.0

With a recent 5.0 release these two products were merged together. There is a single artefact to use - com.hazelcast:hazelcast. You just create a Hazelcast instance and, if enabled, you can get the Jet engine from there using HazelcastInstance#getJet

5.0 is 100 % compatible with IMDG 4.2, just change the dependency and mostly compatible with Jet 4.5, some code changes are needed though.