Search code examples
pythonapache-sparkpysparkpython-packagingpython-wheel

How do I call pyspark code with .whl file?

I have used poetry to create a wheel file. I am running following spark-submit command , but it is not working. I think I am missing something

spark-submit --py-files /path/to/wheel

Please note that I have referred to below as well, but did not get much details as I am new to Python. how to pass python package to spark job and invoke main file from package with arguments

Solution

  • Wheel file can be executed as a part of below spark-submit command

    spark-submit --deploy-mode cluster --py-files /path/to/wheel main_file.py