Search code examples
scalapysparkxgboost

TypeError: 'JavaPackage' object is not callable for XGBoost in PySpark

I am trying to make Scala Xgboost API available for my PySpark Notebook. And following this blog. However, keep on running into the below error:

spark._jvm.ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator
<py4j.java_gateway.JavaPackage at 0x7fa650fe7a58>

from sparkxgb import XGBoostEstimator

xgboost = XGBoostEstimator(
    featuresCol="features", 
    labelCol="Survival", 
    predictionCol="prediction"
)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-1765fb9e3344> in <module>
      4     featuresCol="features",
      5     labelCol="Survival",
----> 6     predictionCol="prediction"
      7 )

~/spark-assembly-2.4.0-twttr-kryo3-scala2128-hadoop2.9.2.t05/python/pyspark/__init__.py in wrapper(self, *args, **kwargs)
    108             raise TypeError("Method %s forces keyword arguments." % func.__name__)
    109         self._input_kwargs = kwargs
--> 110         return func(self, **kwargs)
    111     return wrapper
    112 

~/local/spark-3536cd7a-6188-4ca8-b3d0-57d42cd01531/userFiles-0a0d90bc-96b4-43f2-bf21-00ae0e6f7309/sparkxgb.zip/sparkxgb/xgboost.py in __init__(self, checkpoint_path, checkpointInterval, missing, nthread, nworkers, silent, use_external_memory, baseMarginCol, featuresCol, labelCol, predictionCol, weightCol, base_score, booster, eval_metric, num_class, num_round, objective, seed, alpha, colsample_bytree, colsample_bylevel, eta, gamma, grow_policy, max_bin, max_delta_step, max_depth, min_child_weight, reg_lambda, scale_pos_weight, sketch_eps, subsample, tree_method, normalize_type, rate_drop, sample_type, skip_drop, lambda_bias)
    113 
    114         super(XGBoostEstimator, self).__init__()
--> 115         self._java_obj = self._new_java_obj("ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator", self.uid)
    116         self._create_params_from_java()
    117         self._setDefault(

~/spark-assembly-2.4.0-twttr-kryo3-scala2128-hadoop2.9.2.t05/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
     65             java_obj = getattr(java_obj, name)
     66         java_args = [_py2java(sc, arg) for arg in args]
---> 67         return java_obj(*java_args)
     68 
     69     @staticmethod

TypeError: 'JavaPackage' object is not callable

I already googled this error and tried the below things. I got all the ideas from this blog:

  1. Make sure Xgboost4j is in the SPARK_DIST_CLASSPATH. Already checked.
$echo $SPARK_DIST_CLASSPATH |  tr " " "\n" | grep 'xgboost4j' | rev | cut -d'/' -f1 | rev
xgboost4j-0.72.jar
xgboost4j-spark.72.jar
  1. Make sure they are added to EXTRA_CLASSPATH. - Done
  2. Updating configs.
'export PYSPARK_SUBMIT_ARGS="--conf spark.jars=$SPARK_HOME/jars/* --conf spark.driver.extraClassPath=$SPARK_HOME/jars/* --conf spark.executor.extraClassPath=$SPARK_HOME/jars/* pyspark-shell"',

Hardware Info:

  • Machine: Linux
  • Using Jupyter Notebook.
  • Spark Version 2.4.0
  • python3.6
Solution

  • I found the problem, The problem was that the sparkxbg.zip(which I downloaded over the internet) is written for xgboost4j-0.72. However, my jars were from xgoost4j-0.9. And the API has been completely changed. As a result, the 0.9 version didn't have any class named ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator. And hence the error.

    You can see the difference in API here: release_0.72 vs v0.90