Search code examples
jsonhazelcast-jet

JSON processing using Hazelcast Jet

I'm using PostgreSQL as database.

-------------------------
id   |  name   | value  |
-------------------------
 1   |  john   |  {"id":"0001"}
-------------------------

Above is simple JSON but in the application level JSON is very big. So once we upload json data into jet how to process json data by creating a pipelines in Hazelcast Jet. Any simple suggestions would be helpful to get start on this.

Solution

  • There are two modes how to get data from PostgreSQL to Jet:

    1/ Batch mode, where Jet obtains the data based on the SQL query you provide. See the code sample. This mode is available via JDBC source and produces a ResultSet that you map to your domain object.

    2/ Streaming mode, where Jet subscribes to the changes to the DB tables and gets notified for any INSERT, UPDATE or DELETE. The event contains the new row value. See the tutorial. This mode is available via PostgresCdcSource and produces ChangeRecords.

    Both modes give you the raw JSON as a string. You can use the JSON operator to conveniently map the JSON data to your domain objects so that you can process them by further operators downstreams.