Loop through each subdirectory in a main directory and run code against each file using OS

Question

Essentially what I'm trying to do is loop through a directory that contains multiple sub-directories and within those run code against each file in a for loop.

The only start I managed to make was listing the directories but as I've rarely ever used os I'm not sure if I could potentially loop through os.chdir and a bit of f string formatting to loop through each subdirectory.

The files I want to run code against are just txt files.

Here goes my code, up to the moment:

import os 

for folders in os.listdir('../main_directory'):
    for something in os.listdir(f'{folders}'):
        # run some function of sorts
        pass

Any help would be greatly appreciated.

Answer 1

I like using pure os:

import os

for fname in os.listdir(src):

    # build the path to the folder
    folder_path = os.path.join(src, fname)
    
    if os.path.isdir(folder_path):
        # we are sure this is a folder; now lets iterate it
        for file_name in os.listdir(folder_path):
            file_path = os.path.join(folder_path, file_name)
            # now you can apply any function assuming it is a file
            # or double check it if needed as `os.path.isfile(file_path)`

Note that this function just iterate over the folder given at src and one more level:

src/foo.txt  # this file is ignored
src/foo/a.txt  # this file is processed
src/foo/foo_2/b.txt  # this file is ignored; too deep.
src/foo/foo_2/foo_3/c.txt  # this file is ignored; too deep.

In case you need to go as deep as possible, you can write a recursive function and apply it to every single file, as follows:

import os

def function_over_files(path):
    if os.path.isfile(path):
        # do whatever you need with file at path
    else:
        # this is a dir: we will list all files on it and call recursively
        for fname in os.listdir(path):
            f_path = os.path.join(path, fname)

            # here is the trick: recursive call to the same function
            function_over_files(f_path)

src = "path/to/your/dir"
function_over_files(src)

This way you can apply the function to any file under path, don't care how deep it is in the folder, now:

src/foo.txt  # this file is processed; as each file under src
src/foo/a.txt  # this file is processed
src/foo/foo_2/b.txt  # this file is processed
src/foo/foo_2/foo_3/c.txt  # this file is processed