Search code examples
gitjenkinsgit-cloneshallow-clone

Jenkins not able to pull 20 GB git repo

My Jenkins build is failed as it's not able to pull 20 GB git repo. Please let me know where I am wrong or missing something.

For details please see screenshots below:

Error Log

Jenkins config

Note: I tried increasing time out but it didn't work

Solution

  • A couple of ideas:

    1. Check the exact time period (the time it takes) since jenkins starts pulling the repository till the time it fails. If this time is always the same, there is a chance that there is some network gear that has timeout per request. You might find this plugin useful

    2. Consider using shallow cloning - an option that won't pull the whole repository with the whole history since usually its not required for the build, you need only the last snapshot. So shallow cloning does exactly that and as a result the size of the pulled data is much smaller.

    3. Consider checking out only the stuff that you require. Anyway, try sparse checkouts

    4. If you can access git repository and change some stuff there, consider using git lfs. I know its beyond the scope of the question, but 20 GB repository sounds very suspicious to me, probably its a sign that git wasn't used properly, there are some binary large files in it, and so forth. So cleaning it up/ using git lfs for storing binary files sounds like a great git optimization that alone can resolve all the issues with git pull in jenkins .

    All in all, I believe this PDF might provide you more ideas.