Search code examples
hazelcast-jet

How to use a Twitter stream source in Hazelcast Jet without needing a DAG?

I want to do simple analysis on a live stream of tweets.

How do you use a Twitter stream source in Hazelcast Jet without needing a DAG?

Details

The encapsulation of Twitter API is pretty good at StreamTwitterP.java.

However, the caller uses that as part of a DAG, c/o:

Vertex twitterSource = 
  dag.newVertex("twitter", StreamTwitterP.streamTwitterP(properties, terms));

My use case doesn't need the power of DAG, so I'd rather avoid that needless extra complexity.

To avoid a DAG, I'm looking to use SourceBuilder to define a new data source for live stream of tweets.

I assume that would have code similar to StreamTwitterP.java, mentioned above, however it's not clear to me the fit using the API of Hazelcast JET.

I was referring to SourceBuilder example from the docs.

Solution

  • You can convert a processor to a pipeline source:

    Pipeline p = Pipeline.create();
p.drawFrom(Sources.<String>streamFromProcessor("twitter", 
    streamTwitterP(properties, terms)))
...

    There's also twitterSource version that uses SourceBuilder here.