getting OSError: Passed non-file path using pyarrow ParquetDataset

Question

I have to run some tests on different environments. In tests I have to check some directories in s3 to find parquet files and transfer them to dictionary like this

import pyarrow.parquet as pq
import s3fs

env = 'dev'
aws_profile ={'dev': 'dev_aws_profile', 'qa': 'qa_aws_profile'}
def get_dictionary_from_parquet(file_name):
    fs = s3fs.S3FileSystem()
    pq_session = Session(profile_name=aws_profile[env])
    s3 = pq_session.resource('s3')
    parquet_bucket = s3.Bucket(f'valid-bucket-name-{env}')
    paths = []
    for pq_file in parquet_bucket.objects.filter(Prefix=f'valid-prefix-{env}'):
    if pq_file.key.endswith(file_name):
        paths.append(f's3://{pq_file.bucket_name}/{pq_file.key}')
    data_set = pq.ParquetDataset(paths, filesystem=fs)
    tbl = data_set.read()
    pq_dictionary = tbl.to_pydict()
    return pq_dictionary

it works perfectly if aws_profile == default profile in aws credentials file, but it returns

line 14, in get_dictionary_from_parquet
    data_set = pq.ParquetDataset(paths, filesystem=fs)
  File "/Library/Python/3.7/site-packages/pyarrow/parquet.py", line 1170, in __init__
    open_file_func=partial(_open_dataset_file, self._metadata)
  File "/Library/Python/3.7/site-packages/pyarrow/parquet.py", line 1365, in _make_manifest
    .format(path))    
OSError: Passed non-file path: s3://<valid path to parquet file>

how to parse aws profile creds to pyarrow to fix it?

Answer 1

It is weird that you are configuring and doing your file filtering on boto(3?) object, while using the s3fs instance to specify the filesystem when reading. I recommend using s3fs for both.

The following will fix it

fs = s3fs.S3FileSystem(profile=aws_profile[env])

but I would suggest that you can use the same instance to do your file listing too

paths = fs.glob(f"valid-bucket-name-{env}/valid-prefix-{env}/*/file_name")

(or whatever the right glob pattern is - I had trouble parsing your code).