How to import text file in Data bricks

Question

I am trying to write text file with some text and loading same text file in data-bricks but i am getting error

Code

#write a file to DBFS using Python I/O APIs
with open("/dbfs/FileStore/tables/test_dbfs.txt", 'w') as f:
  f.write("Apache Spark is awesome!\n")
  f.write("End of example!")

# read the file
with open("/dbfs/tmp/test_dbfs.txt", "r") as f_read:
  for line in f_read:
    print(line)

Error FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/tables/test_dbfs.txt'

Answer 1

The /dbfs mount doesn't work on Community Edition with DBR >= 7.x - it's a known limitation.

To workaround this limitation you need to work with files on the driver node and upload or download files using the dbutils.fs.cp command (docs). So your writing will look as following:

#write a file to local filesystem using Python I/O APIs
with open("'file:/tmp/local-path'", 'w') as f:
  f.write("Apache Spark is awesome!\n")
  f.write("End of example!")
# upload file to DBFS
dbutils.fs.cp('file:/tmp/local-path', 'dbfs:/FileStore/tables/test_dbfs.txt')

and reading from DBFS will look as following:

# copy file from DBFS to local file_system
dbutils.fs.cp('dbfs:/tmp/test_dbfs.txt', 'file:/tmp/local-path')
# read the file locally
with open("/tmp/local-path", "r") as f_read:
  for line in f_read:
    print(line)