Search code examples
parametersstreamingsnappydata

SnappyData -streaming table parameter "topics" clarification

I was reading through the documentation (http://snappydatainc.github.io/snappydata/streamingWithSQL/) and wanted to know what the parameter ":01" means after the topic name when working with kafka. Is this a partition number or number of threads server is using?

ie: "topics 'streamTopic:01'

val sc = new SparkContext(new SparkConf().setAppName("example").setMaster("local[*]"))
val snc = SnappyContext.getOrCreate(sc)
var snsc = SnappyStreamingContext(snc, Seconds(1))

snsc.sql("create stream table streamTable (userId string, clickStreamLog string) " +
    "using kafka_stream options (" +
    "storagelevel 'MEMORY_AND_DISK_SER_2', " +
    "rowConverter 'io.snappydata.app.streaming.KafkaStreamToRowsConverter', " +
    "zkQuorum 'localhost:2181', " +
    "groupId 'streamConsumer', " +
    "topics 'streamTopic:01')")

Sorry if this is mentioned somewhere, but I could not find it.

Solution

  • This should be the topic names only. Spark streaming will figure out the number of kafka partitions and start enough parallel tasks for ingestion.

    See another example here.