Connecting to Data Streams for Memgraph Import

Question

I need to decide which streaming technology we should implement for our next project. I can choose between Kafka, Redpanda, or Pulsar stream, and I need to import streams into Memgraph. How can I connect directly to these streams using Cypher queries or Memgraph Lab to import the data efficiently?

Answer 1

Memgraph supports all three sources that you mention: Apache Kafka, Redpanda and Pulsar.

Importing data from streams is four step process:

1. Start Memgraph and connect to the database
2. Create and load a transformation module into Memgraph
3. Create a stream in Memgraph
4. Start ingesting data from the stream

There are some differences between configuration depending on the data source that you use. Detailed instructions and Cypher queries can be found at https://memgraph.com/docs/memgraph/import-data/data-streams/manage-streams.

If you are using Memgraph Lab take a look at official documentation at https://memgraph.com/docs/memgraph/import-data/data-streams/manage-streams-lab.