Is there a way to use the filename in a dataset and have it be the column.
ie if the directory has
file1.parquet file2.parquet file3.parquet
can loading that as a dataset then have a column with the values file1, file2, and file3?
or does it only work with directory names? It seems to only work with directory names, is that right?
Support for filename-based partitioning will be in Arrow 8.0.0, which will likely release later this month or in May 2022. See ARROW-14612. The same goes for being able to have a column with the filename, see ARROW-15281.