Search code examples
pythonpandaspysparkpalantir-foundryspark-koalas

How do you import and use Spark-Koalas in palantir-foundry


How can I -- in Palantir-foundry -- import and use the "Koalas: pandas API for Apache Spark" open source python package.

I know that you can import packages that don't exist through Code Repo and have done this, can I do this same process for Koalas package or do I need to follow another route?


Solution

  • I was able to use Code Repo to upload a local clone of the package and then add the package in platform using the steps detailed here: How to create python libraries and how to import it in palantir foundry

    However, shortly afterwards Palantir admins introduced an update which included the Koalas package as a native package to the platform. I have not however had time to try using this for any major tasks as of yet.