Search code examples
apache-sparkapache-kafkaspark-streamingapache-kafka-streams

Can we use Kstream with Spark Streaming?

Can we use Kstream with Spark?

Is it a recommended approach, or using Spark Streaming is a better solution?

Solution

  • Can we use Kstream with Spark?

    Sure.

    Is it a recommended approach

    Not really (if at all).

    using the Spark Streaming is a better solution.

    "Better" begs for another question "In what scenarios?"

    Kafka Streams is a library and as such can be used anywhere a library could be used in an application, incl. Spark applications. In that sense, it is possible, but not really of much help IMHO.

    Kafka Streams is simply a bunch of threads that use Consumer and Producer APIs to transform records. That's the Kafka data format in Spark Structured Streaming exactly.

    Kafka Streams applications are deployed as standalone Java applications (e.g. Docker containers in k8s). The same is also possible with Spark Structured Streaming (with Spark on Kubernetes).

    See no real benefit using both in a single application but would love proven wrong.