I implemented in java a spark streaming job following the instructions at https://spark.apache.org/docs/2.1.0/streaming-kafka-0-10-integration.html. It works perfectly when Kafka and Cassandra servers are standalone. Extract of logs
...
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
18/02/04 20:40:35 INFO AppInfoParser: Kafka version : 0.11.0.0
18/02/04 20:40:35 INFO AppInfoParser: Kafka commitId : cb8625948210849f
18/02/04 20:40:35 INFO AbstractCoordinator: Discovered coordinator spark-vm:9092 (id: 2147483647 rack: null) for group group_1.
18/02/04 20:40:35 INFO ConsumerCoordinator: Revoking previously assigned partitions [] for group group_1
18/02/04 20:40:35 INFO AbstractCoordinator: (Re-)joining group group_1
[2018-02-04 20:40:35,929] INFO [GroupCoordinator 0]: Preparing to rebalance group group_1 with old generation 6 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
[2018-02-04 20:40:35,946] INFO [GroupCoordinator 0]: Stabilized group group_1 generation 7 (__consumer_offsets-49) (kafka.coordinator.group.GroupCoordinator)
...
The problem appears when I deploy in a mesos cluster since the job connects to kafka but it doesnt discover any coordinator
metrics.sample.window.ms = 30000
auto.offset.reset = latest
18/02/04 19:27:41 INFO AppInfoParser: Kafka version : 0.10.0.1
18/02/04 19:27:41 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5
18/02/04 19:27:42 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_RUNNING
18/02/04 19:27:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.65.111:50842) with ID 1
18/02/04 19:27:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.65.111:39485 with 366.3 MB RAM, BlockManagerId(1, 192.168.65.111, 39485, None)
18/02/04 19:27:47 INFO CassandraConnector: Disconnected from Cassandra cluster: cassandra
2018-02-04 19:32:19,161:7(0x7fb4fd02b700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 12ms
ZK used by Kafka is master.mesos:2181/dcos-service-kafka but I dont see how to configure it in Spark job and I'm not even sure this is the problem or if it is just a kafka versions issue.
Problem was connectivity one. Spark job was not able to connect to Kafka using the VIP endpoint provided by DC/OS but the logs didn't throw any exception as a someone new in these technologies would expect.