Search code examples
databricksdbutils

java.io.FileNotFoundException on local databricks directory but directory exists

I am trying to copy a directory in databricks from one location to another

dbutils.fs.cp("/path/to/directory", "new_path/to/directory")

Prior to running this I confirm the directory exists:

folders <- list.folders("/path/to/", full.names = TRUE) 
print(folders)

I receive this error despite /path/to/directory being confirmed as existing:

java.io.FileNotFoundException: /path/to/directory
Solution

  • If you don't specify the schema, then it's assumed dbfs:, while you're trying to copy local file to DBFS. You need to specify file: schema when working with local files:

    To copy local file to DBFS:

    dbutils.fs.cp("file:/path/to/directory", "new_path/to/directory")