CreatePairedStream is not a member of MQTTUtils

Question

When I declared MQTTUTils.createPairedStream() I got an error like

value createPairedStream is not a member of object org.apache.spark.streaming.mqtt.MQTTUtils

My spark and Scala versions are

SCALA VERSION - 2.11.8

SPARK VERSION - 2.3.0

Answer 1

You can see the following pull request in apache Bahir: Bahir Pull Request

Where you can see MQTTUtils.createPairedStream being added.

You import in your pom/gradle/sbt... using the following artifact:

spark-sql-streaming-mqtt_2.11 version 2.3.2 from the group org.apache.bahir.

Up to Spark 1.6 you could use in maven:

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-mqtt -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-mqtt_2.11</artifactId>
    <version>1.6.3</version>
</dependency>

For Spark 2.3.2, you'll need to use:

<dependency>
    <groupId>org.apache.bahir</groupId>
    <artifactId>spark-streaming-mqtt_2.11</artifactId>
    <version>2.3.2</version>
</dependency>

Or in SBT:

libraryDependencies += "org.apache.bahir" %% "spark-streaming-mqtt" % "2.3.2"

You can find further information on: org.apache.bahir:spark-streaming-mqtt

bin/spark-shell --packages org.apache.bahir:spark-streaming-mqtt_2.11:2.3.0

You will import the package using scala:

import org.apache.spark.streaming.mqtt._

and instantiate:

val lines = MQTTUtils.createPairedStream(ssc, brokerUrl, topic)

I hope this helps.