How to push a large folder onto github, ignoring all files over 100MB?

Question

I have an enormously large folder. In this folder are a lot of videos and small files scattered around. I value the small files, not the videos.

I want to back up all the small files onto Github. If I directly try to upload the folder onto Github, I will get multiple large fille (>100 MB files) errors. The videos and small files are scattered around everywhere, making it difficult to remove videos one by one.

I know how to remove a single giant file from Github: Git error, need to remove large file

However, is there an easy way to do this for all the large files in my folder, such that I can push everything else onto Github with ease?

Note: I don't want to use cloud storage, because all I want to store are my small files.

Answer 1

If you want to retain these large files locally, you first delete them from Git, but retain them in the working copy.

1. Find them all with find . -size +100M -not -path './.git/*'
2. Remove the files from Git, but keep the file itself: git rm --cached.
3. Add them to .gitignore.

You can combine steps 1 and 2: find -X . -size +100M -not -path './.git/*' | xargs git rm --cached.

I would recommend using this as an opportunity to move all your large assets to a single directory and ignore that directory.

Now that they're deleted from Git and safely ignored, use the BFG Repo-Cleaner to delete them from history.

bfg --strip-blobs-bigger-than 100M --no-blob-protection my-repo.git