Search code examples
databricksdatabricks-community-edition

Unable to access file from DBFS using DataBricks fuse mount path

I have files in data bricks as shown below enter image description here

I am trying to access them like this from data bricks notebooks enter image description here

But I am getting error, even trying to use pandas gives an error enter image description here

I don't understand where am I going wrong. Althought dbutils.fs.head('/FileStore/tables/flights_small.csv') gives me result correctly.

Solution

  • You are using DataBricks Community Edition, because of a quirk with DBR >= 7.0, you cannot read in from your path.

    I usually just have a command like the new one below to resolve this issue and programmatically bring te file to the accessible temp folder:

    %fs cp /FileStore/tables/flights_small.csv file:/tmp/flights_small.csv

    then simply read it in:

    pd.read_csv('file:/tmp/flights_small.csv')

    Given quirks like this in the Community Edition (long cluster startup times), I usually go with Google Colab for hobby work in a browser notebook.

    Also you can run pyspark on Google Colab with just

    !pip install pyspark

from pyspark.sql import SparkSession
spark = SparkSession.builder\
        .master("local")\
        .appName("Colab")\
        .config('spark.ui.port', '4050')\
        .getOrCreate()