Search code examples
apache-sparkipythonzeromqapache-zeppelinakka-zeromq

Which protocol is Apache Zeppelin using to connect to Apache Spark? (iPython/ZeroMQ)?

I'm wondring which protocol Apache Zeppelin is using to connect to Apache Spark?

Is it also the iPython Protocol on ZeroMQ?

Thanks a lot!

best regards

Romeo

Solution

  • Spark Interpreter for Apache Zeppelin (incubating) actually uses neither of them.

    It just hosts Spark Driver inside spark interpreter process and submits a job per paragraph though SparkIMain, in the same way as spark-shell does.

    I believe an internal low-level communication between Driver and Master is based on Akka but is a Spark-specific implementation detail.