Search code examples
apache-sparkapache-kafkaapache-spark-sqlspark-streaming

Version of Kafka Connector For Use in Spark Streaming

The latest version of Kafka available for download is Kafka 2.1.0. But in order to use Kafka in Spark Streaming, or Spark Structured Streaming, we use respectively the following connectors:

spark-streaming-kafka-0-10_2.11
spark-sql-kafka-0-10_2.11

My question is that it seems that the connectors are for Kafka version 0.10.0.0 since the name of the connectors include 0-10. Is there something that I don't understand here, or we are really using connectors which are for much older versions of Kafka?

Solution

  • For Spark Structure Streaming 2.4, Kafka Client 2.0 is used.

    0-10 means it is compatible with Kafka Brokers in version 0.10 or above.

    You can check it in pom.xml in spark project: https://github.com/apache/spark/blob/branch-2.4/external/kafka-0-10-sql/pom.xml#L33