Search code examples
pythonapache-sparkpysparkpycharmdatabricks

How can I use databricks utils functions in PyCharm? I can't find appropriate pip package

PyCharm IDE. I want to use dbutils.widgets.get() in a module and than to import this module to databricks. I already tried with pip install databricks-client pip install databricks-utils and pip install DBUtils

Solution

  • The dbutils is available only as a part of the databricks-connect package. Its documentation contains detailed description on how to setup PyCharm to work with it. It also covers on how to use the dbutils.

    You may need to define following wrapper to be able to use dbutils locally and on Databricks:

    def get_dbutils(spark):
  from pyspark.dbutils import DBUtils
  return DBUtils(spark)

get_dbutils().fs.cp('file:/home/user/data.csv', 'dbfs:/uploads')