Python \ PySpark code: Isn't this an infinite loop?

Question

I was looking through the PySpark code for class pyspark.mllib.feature.IDFModel

at

http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.feature.IDFModel.transform

I am confused why this does not create an infinite loop?

This is the code in the class:

class IDFModel(JavaVectorTransformer):

  def transform(self, x):
    return JavaVectorTransformer.transform(self, x)

and this is the code for the JavaVectorTransformer Class

class JavaVectorTransformer(JavaModelWrapper, VectorTransformer):

  def transform(self, vector):
    if isinstance(vector, RDD):
        vector = vector.map(_convert_to_vector)
    else:
        vector = _convert_to_vector(vector)
    return self.call("transform", vector)

IDFModel.transform() calls JavaVectorTransformer.transform which in turn returns self.call("transform",vector)

Strangely this seems to work without creating an infinite loop.

Any thoughts?

Answer 1

Refer the code for JavaModelWrapper in the link https://spark.apache.org/docs/1.5.0/api/python/_modules/pyspark/mllib/common.html

self.call is not a recursive call to the same method. There is a method called "call" in the parent class JavaModelWrapper that gets called when self.call is invoked.