Search code examples
gitjenkinsdynamicnas

How to fix Jenkins job that is failing on git checkout when workspace is on network share, but not when workspace is on local drive?


I'm building a service that creates dynamic nodes that will connect to a Jenkins server. The nodes will be used primarily for building many large code bases, so I would like to take advantage of incremental builds. Since the nodes are ephemeral virtual machines that get destroyed sometime after the job is complete, I need the Jenkins workspace to be something other than a local drive on the machine.

I created a network share and I set the root directory for the Jenkins node to I:\jenkins. When the VM comes online, my service executes a command that maps the I drive to the network share. The service then executes commands to connect the service to the Jenkins master. When I attempt to run a build against the node, it consistently fails when trying to git checkout.

After the job fails, the network share has most of the files from the git repository, so the share is accessible and there's no read/write restrictions preventing the job from running. This error happens with any job I try to run, including jobs that access git repos with very few files.

The log output in the failed Jenkins job is:

git checkout -f ...a70f FATAL: Could not checkout ...a70f hudson.plugins.git.GitException: Could not checkout ...a70f ...<stack trace>... stderr: error: failed to read object ...f656 at offset ...1596 from .git/objects/pack/pack-...2475.pack fatal: packed object ...f656 (stored in .git/objects/pack/pack-...2475.pack) is corrupt

If I change the workspace for the node to a local drive and reconnect the exact same machine and re-run the exact same job, I never see a failure. Git properly clones everything I need, and my job goes green.

How can I fix the issue I'm seeing with the network share? Or alternatively, how can I meet the following requirements:

  • The Jenkins nodes must run on ephemeral VMs that might be destroyed and recreated at any time
  • The workspace must be visible to multiple VMs concurrently (stringent workspace naming for the jobs prevents multiple nodes from reading/writing the same files)
  • The workspace must persist even if there are no Jenkins nodes online
  • Any new nodes that come online must be able to reconnect to the workspace

If there's a solution that doesn't require a network share, I would be happy to try something else!


Solution

  • Found a solution here: GIT pull error - remote object is corrupted

    For some reason the default git config values work perfectly fine when the workspace is in local storage. When moving to any externally mapped storage solution the git config needed to be modified to prevent