Search code examples
linuxbashshellgnu-findutils

Find the name of subdirectories and process files in each

Let's say /tmp has subdirectories /test1, /test2, /test3 and so on, and each has multiple files inside.

I have to run a while loop or for loop to find the name of the directories (in this case /test1, /test2, ...) and run a command that processes all the files inside of each directory.

So, for example, I have to get the directory names under /tmp which will be test1, test2, ... For each subdirectory, I have to process the files inside of it.

How can I do this?

Clarification:

This is the command that I want to run:

find /PROD/140725_D0/ -name "*.json" -exec /tmp/test.py {} \;

where 140725_D0 is an example of one subdirectory to process - there are multiples, with different names.

So, by using a for or while loop, I want to find all subdirectories and run a command on the files in each.

The for or while loop should iteratively replace the hard-coded name 140725_D0 in the find command above.

Solution

  • You should be able to do with a single find command with an embedded shell command:

    find /PROD -type d -execdir sh -c 'for f in *.json; do /tmp/test.py "$f"; done' \;

    Note: -execdir is not POSIX-compliant, but the BSD (OSX) and GNU (Linux) versions of find support it; see below for a POSIX alternative.

    • The approach is to let find match directories, and then, in each matched directory, execute a shell with a file-processing loop (sh -c '<shellCmd>').
    • If not all subdirectories are guaranteed to have *.json files, change the shell command to for f in *.json; do [ -f "$f" ] && /tmp/test.py "$f"; done

    Update: Two more considerations; tip of the hat to kenorb's answer:

    • By default, find processes the entire subtree of the input directory. To limit matching to immediate subdirectories, use -maxdepth 1[1]:

      find /PROD -maxdepth 1 -type d ...

    • As stated, -execdir - which runs the command passed to it in the directory currently being processed - is not POSIX compliant; you can work around this by using -exec instead and by including a cd command with the directory path at hand ({}) in the shell command:

      find /PROD -type d -exec sh -c 'cd "{}" && for f in *.json; do /tmp/test.py "$f"; done' \;

    [1] Strictly speaking, you can place the -maxdepth option anywhere after the input file paths on the find command line - as an option, it is not positional. However, GNU find will issue a warning unless you place it before tests (such as -type) and actions (such as -exec).