Search code examples
pythonpandasglob

Using glob recursion to get sub directories and files containing CSVs

I am trying to concat multiple CSVs that live in subfolders of my parent directory.

/ParentDirectory
│  
│
├───SubFolder 1
│       test1.csv
│
├───SubFolder 2
│       test2.csv
│
├───SubFolder 3
│       test3.csv
│       test4.csv
│
├───SubFolder 4
│       test5.csv

When I do

import pandas as pd
import glob

files = glob.glob('/ParentDirectory/*.csv', recursive=True)
df = pd.concat([pd.read_csv(fp) for fp in files], ignore_index=True)

I get ValueError: No objects to concatenate.

But if I select a specific sub folder, it works:

files = glob.glob('/ParentDirectory/SubFolder 3/*.csv', recursive=True)

How come glob isn't able to go down a directory and get the CSVs within each folder of the parent directory?

Solution

  • Try:

    files = glob.glob('/ParentDirectory/**/*.csv', recursive=True)