Search code examples
apache-sparkkubernetesapache-zeppelin

Does Zeppelin 0.10.0 try to run interpreter in k8s cluster?

version
Java 1.8.0_242-b08
Spark 2.4.5
Zeppelin 0.10.0

Inside a k8s Pod, with above set, web ui running, I opened Zeppelin's spark interpreter and ran sc. Following Error prints out:

io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create] for kind: [Pod] with name: [null] in namespace: [default] failed.

Besides that it's not working, what I don't understand is whether Zeppelin use k8s and how.

Did some searching. fabric8 seems to be related to k8s and I suppose Zeppelin tries to run its interpreters in k8s cluster. Yet, k8s is not installed (?... like kubectl?) in the machine (where Zeppelin is running). So there are two explanation I can think of:

  1. fabric8 include k8s so Zeppelin creates local k8s cluster to run its interpreters.
  2. Zeppelin somehow detected it's been run in k8s Pod and tries to use that Cluster to create Pod, run interpreters in it.

Think the first explanation is the correct one. But running local k8s cluster under the hood feels... how? Error itself was solved by using different versions: Spark 2.4.0, Zeppelin 0.8.2, yet would be nice if I could use up-to-date Spark and Zeppelin.

And in addition, the error itself seems to be related with specific Java version.

Solution

  • Found out Zeppelin has a parameter ZEPPELIN_RUN_MODE. By default it is set as "auto". If you are running Zeppelin in a Pod and don't want to run within k8s sidecar, you should set it "local".