I am totally new to Spark, My spark master is on GCP Kubernetes Standard cluster. I am using a gremlin client to trigger the spark job, on my local machine configuration it works totally fine and the job gets completed, But on Kubernetes, I am seeing the below error.
Gremlin Query and error
==>tinkerpop.hadoop activated
gremlin> :plugin use tinkerpop.spark
==>tinkerpop.spark activated
gremlin> graph = GraphFactory.open('/opt/janusgraph-0.6.2/conf/hadoop-graph/read-hbase-standalone-cluster-stg-bigtable-spark.properties')
==>hadoopgraph[hbaseinputformat->nulloutputformat]
gremlin>
gremlin> g = graph.traversal().withComputer(SparkGraphComputer)
==>graphtraversalsource[hadoopgraph[hbaseinputformat->nulloutputformat], sparkgraphcomputer]
gremlin> g.V().count()
14:01:48 WARN org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer - class org.apache.hadoop.mapreduce.lib.output.NullOutputFormat does not implement PersistResultGraphAware and thus, persistence options are unknown -- assuming all options are possible
14:02:03 WARN org.apache.spark.scheduler.TaskSetManager - Lost task 6.0 in stage 0.0 (TID 6, XX.XX.XX.XX, executor 0): java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2198)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedSerializationConstructorAccessor270.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:963)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2195)
... 77 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator
at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
... 81 more
14:02:03 ERROR org.apache.spark.scheduler.TaskSetManager - Task 8 in stage 0.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 0.0 failed 4 times, most recent failure: Lost task 8.3 in stage 0.0 (TID 32, 10.12.11.7, executor 1): java.io.InvalidClassException: org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration; unable to create instance
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2198)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:2117)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1658)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:488)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2340)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2479)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2373)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2211)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:488)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:446)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedSerializationConstructorAccessor238.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:963)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2195)
... 76 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.configuration2.interpol.ConfigurationInterpolator
at org.apache.commons.configuration2.AbstractConfiguration.installDefaultInterpolator(AbstractConfiguration.java:455)
at org.apache.commons.configuration2.AbstractConfiguration.<init>(AbstractConfiguration.java:132)
... 80 more
Config File--> read-hbase-standalone-cluster-stg-bigtable-spark.properties
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Hadoop Graph Configuration
#
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.hbase.HBaseInputFormat
gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat
gremlin.hadoop.jarsInDistributedCache=true
gremlin.hadoop.inputLocation=none
gremlin.hadoop.outputLocation=output
gremlin.spark.persistContext=true
#
# JanusGraph HBase InputFormat configuration
#
#janusgraphmr.ioformat.conf.storage.backend=hbase
#janusgraphmr.ioformat.conf.storage.hostname=localhost
#janusgraphmr.ioformat.conf.storage.port=8586
#janusgraphmr.ioformat.conf.storage.hbase.table=janusgraph
janusgraphmr.ioformat.conf.storage.backend=hbase
janusgraphmr.ioformat.conf.storage.hbase.ext.hbase.client.connection.impl=com.google.cloud.bigtable.hbase2_x.BigtableConnection
janusgraphmr.ioformat.conf.storage.hbase.ext.google.bigtable.project.id=**********
janusgraphmr.ioformat.conf.storage.hbase.ext.google.bigtable.instance.id=*********
janusgraphmr.ioformat.conf.storage.hbase.table=**********
janusgraphmr.ioformat.conf.storage.hbase.ext.hbase.regionsizecalculator.enable=false
# This defines the indexing backend configuration used while writing data to JanusGraph.
janusgraphmr.ioformat.conf.index.search.backend=elasticsearch
janusgraphmr.ioformat.conf.index.search.hostname=**********
#
# SparkGraphComputer Configuration
#
spark.master=spark://spark-master:7077
spark.executor.memory=1g
spark.executor.extraClassPath=/opt/janusgraph-0.6.2/lib/*
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator```
With reference to this https://li-boxuan.medium.com/configure-janusgraph-0-6-0-for-spark-64bd7628fb77 blog. spark.executor.extraClassPath=/opt/janusgraph-0.6.2/lib/* mentioned in the gremlin reference config files, these libraries should be present in both Master and Worker nodes.