Search code examples
pysparkdatabricksaws-databricks

Databricks write config file to dbfs:/


Is it possible to write/modify (aka vim) file at dbfs:/ using databricks-cli or UI? I don't have an option to write this file from the notebook directly.

Example: I need to create .ini file, put properties here, and later read them from a notebook.


Solution

  • The easiest way is to use the databricks-cli copy command to put the configuration file there. This way you can also fetch existing files, modify it and move it back to DBFS.

    I don't exactly know your use case, but that is the most straightforward way of solving this. At my company, this is our standard approach for shipping configurations files to Databricks (we do this from the CI/CD).

    Alternatively, dbx might be interesting to look at. It's based around the databricks-cli and makes shipping your project and dependencies easier. Since I don't know your use case, I thought it was worth mentioning.