Python - Navigating through Subdirectories that Meet Naming Criteria

Question

I am using Python 3.5 to analyze data contained in csv files. These files are contained in a "figs" directory, which is contained in a case directory, which is contained in an overall data directory, e.g.:

/strm1/serino/DATA/06052009/figs

Or more generally:

/strm1/serino/DATA/case_date_in_MMDDYYYY/figs

The directory I am starting in is '/strm1/serino/DATA/,' and each subdirectory is the month, day, and year of a case I am working with. Each subdirectory contains another subdirectory named 'figs,' and that is the location of each case's csv file. To be exact:

/strm1/serino/DATA/case_date_in_MMDDYYYY/figs/case_date_in_MMDDYYYY.csv

So, I would like to start in my DATA directory and go through its subdirectories to find those that have the MMDDYYYY naming. However, some of the case directories may be named with a state abbreviation at the end, like: '06052009_TX.' Therefore, instead of matching the MMDDYYYY naming exactly, it could be something as simple as verifying that the directory name contains any number 1 through 9.

Once I am in the first subdirectory (the case directory) I would like to move into the 'figs' subdirectory. Once there, I want to access the csv file with the same naming convention as the first subdirectory (the case directory). I will fill existing arrays with the data contained in each csv file.

Basically, my question concerns navigating through multiple subdirectories that match a certain naming convention and ultimately accessing the data file at the "end." I was naively playing around with glob, fnmatch, os.listdir, and os.walk, but I could not get anything close enough to working that I feel would be helpful to include. I am not very familiar with those modules. What I can include is what I am going for:

    for dirs in data_dir that contain a number:
        go into this directory
        go into 'figs' directory
        read data from the csv file whose name matches its case directory name (or whose name format matches the case directory name format)

I have come across related questions, but I have not been able to apply their answers in the way that I would like, especially with nested directories. I really appreciate the help, and let me know if I need to clarify anything.

Answer 1

The following should get you going. It uses the datetime.strptime() function to attempt to convert each folder name into a valid datetime object. If the conversion fails, then you know that the folder name is not in the correct format and can be skipped. It then attempts to parse any CSV file found in the corresponding fig folder:

from datetime import datetime
import glob
import csv
import os

dirpath, dirnames, filenames = next(os.walk('/strm1/serino/DATA'))

for dirname in dirnames:
    if len(dirname) >= 8:
        try:
            dt = datetime.strptime(dirname[:8], '%m%d%Y')
            print(dt, dirname)
            csv_folder = os.path.join(dirpath, dirname)

            for csv_file in glob.glob(os.path.join(csv_folder, 'figs', '*.csv')):
                with open(csv_file, newline='') as f_input:
                    csv_input = csv.reader(f_input)

                    for row in csv_input:
                        print(row)

        except ValueError as e:
            pass