I am working on a Data Proc Spark cluster with an initialization action to install Jupyter notebook. I am unable to read the csv files stored on the google cloud storage bucket, however I am able to read the same files when I work on Spark Shell
Below is the error code I am getting
import pandas as pd
import numpy as np
data = pd.read_csv("gs://dataproc-78r5fe64b-a56d-4f5f4-bcf9-e1b7t6fb9d8f-au-southeast1/notebooks/datafile.csv")
FileNotFoundError Traceback (most recent call last)
<ipython-input-20-2457012764fa> in <module>
----> 1 data = pd.read_csv("gs://dataproc-78r5fe64b-a56d-4f5f4-bcf9-e1b7t6fb9d8f-au-southeast1/notebooks/datafile.csv")
/opt/conda/lib/python3.6/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
676 skip_blank_lines=skip_blank_lines)
677
--> 678 return _read(filepath_or_buffer, kwds)
679
680 parser_f.__name__ = name
/opt/conda/lib/python3.6/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
438
439 # Create the parser.
--> 440 parser = TextFileReader(filepath_or_buffer, **kwds)
441
442 if chunksize or iterator:
/opt/conda/lib/python3.6/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
785 self.options['has_index_names'] = kwds['has_index_names']
786
--> 787 self._make_engine(self.engine)
788
789 def close(self):
/opt/conda/lib/python3.6/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
1012 def _make_engine(self, engine='c'):
1013 if engine == 'c':
-> 1014 self._engine = CParserWrapper(self.f, **self.options)
1015 else:
1016 if engine == 'python':
/opt/conda/lib/python3.6/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
1706 kwds['usecols'] = self.usecols
1707
-> 1708 self._reader = parsers.TextReader(src, **kwds)
1709
1710 passed_names = self.names is None
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()
FileNotFoundError: File b'gs://dataproc-78r5fe64b-a56d-4f5f4-bcf9-e1b7t6fb9d8f-au-southeast1/notebooks/datafile.csv' does not exist
Location path for the CSV file
gs://dataproc-78r5fe64b-a56d-4f5f4-bcf9-e1b7t6fb9d8f-au-southeast1/notebooks/datafile.csv
I have also made sure that the csv file is stored in the same storage bucket as attached to the data proc and have made sure that the file is in UTF-8-Encoded csv format
Can anyone please help me how to read the files stored in google bucket from jupyter notebook running on a dataproc cluster in google cloud.
Kindly let me know if more information is required
Thanks in Advance!!
The reason Spark can read from GCS is that we configure it to use the GCS connector for paths that start with gs://. You probably want to use spark.read.csv(gs://path/to/files/) to read CSV file(s) into a Spark dataframe.
You can read and write to GCS using pandas, but it's a bit more complicated. This stackoverflow post lists some options.
Side note: If you're using Pandas, you should use a single node cluster since pandas code will not be distributed across a cluster.