Search code examples
javaapache-beamgoogle-cloud-platformcometd

Can I read stream data into Apache Beam from a cometd topic using java?

I have a producer spitting messages out on a cometd topic. I need to pick and process the stream of messages from this topic. I have probably spent last two hours trying to find a way to ingest the messages on cometd directly into apache Beam. I seem to be hitting a wall here. I know that I have following options:

  1. Get from topic, Write the data to kafka and then push the data from kafka to beam.
  2. Get from topic, Write the data to pub/sub on GCP and then push it through to Apache Beam.

Both the options above seem to be including an extra component in the architecture. Is there is a better way to do this? Any examples? Code Samples? Pointers?

Solution

  • I am not aware of anybody yet having written or started a CometD connector for Beam. The current connectors are listed at https://beam.apache.org/documentation/io/built-in/. To write your own, you could try mimicking the code of one of the other basic streaming connectors, such as AMQP, MQTT or JMS (Kafka and Pubsub are very advanced and I don't advise guiding your implementation by their source code).