"git rm -r" with globbing

Question

I'm trying to use git rm as part of the filter-branch. I want to delete directories */*/dir1 but nothing is happening. The command I'm using is:

git filter-branch --tag-name-filter cat --index-filter "git rm -r --cached --ignore-unmatch */*/dir1" --prune-empty -f -- --all

Command

find . -wholename "*/*/dir1"

shows a lot of matches.

How I could remove those directories?

Update: I'm trying to remove these folders from the history.

Solution:

With combination of the answers from @j6t and @Sam Gleske here is my solution:

git filter-branch --tag-name-filter cat \
   --index-filter 'git ls-files | grep -E "^([^/]+/){2,3}dir1" |
                   xargs -I {} --no-run-if-empty git rm -r --cached "{}"' \
   --prune-empty -f -- --all

Answer 1

With combination of the answers from @j6t and @Sam Gleske here is my solution:

git filter-branch --tag-name-filter cat \
   --index-filter 'git ls-files | grep -E "^([^/]+/){2,3}dir1" |
                   xargs -I {} --no-run-if-empty git rm -r --cached "{}"' \
   --prune-empty -f -- --all