Search code examples
pythonapache-sparkredispysparkredis-py

NameError: name 'redis' is not defined - PySpark - Redis

I am using the addPyFile method in pyspark to load the redis.zip file. I am able to load the file using

sc.addPyFile("/home/path/to/redis.zip")

But while running the code using ./pyspark, it is showing the error:

NameError: name 'redis' is not defined

The zip(redis.zip) contains .py files(client.py, connection.py,exceptions.py, lock.py,utils.py and others).

Python version is - 3.5 and spark is 2.7

Solution

  • If you pack py files into zip and add it using sc.addPyFile you should import modules using import client, import connector, etc.