Search code examples
javagitgerritrandomaccessfile

git gc appearing to overwrite a packfile leaving open file descriptor holding reference to "old-xxx.pack"

During debugging of a production issue, I am dealing with an application (Gerrit) that holds references to RandomAccessFiles in a cache structure.

These files are referencing a git repositories packfiles.

During an out of band git gc (not within the application) on a repository with no changes, it appears that:

  1. the same packfile is rewritten (same uuid);
  2. file descriptor is in the output list of lsof in the form (old-xxx.pack) but is instantly mark (deleted).

I have been searching numerous codebases for this rename to no avail.

My question is, could this be a filesystem quirk, if a rename/overwrite is done to a file with an open file descriptor by git gc?

lsof entry:

java    25700 user  246r      REG                8,3      5090   1855936 ~/gerrit/git/RepoF.git/objects/pack/old-f4b8054434be1a227bdf8b12729efc1719f39708.pack (deleted)

inotify events during a git gc:

~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ ACCESS pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ ACCESS pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ CREATE tmp_pack_rG4dBh
~/gerrit/git/RepoF.git/objects/pack/ OPEN tmp_pack_rG4dBh
~/gerrit/git/RepoF.git/objects/pack/ MODIFY tmp_pack_rG4dBh
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_WRITE,CLOSE tmp_pack_rG4dBh
~/gerrit/git/RepoF.git/objects/pack/ CREATE tmp_idx_NfF4Jh
~/gerrit/git/RepoF.git/objects/pack/ OPEN tmp_idx_NfF4Jh
~/gerrit/git/RepoF.git/objects/pack/ MODIFY tmp_idx_NfF4Jh
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_WRITE,CLOSE tmp_idx_NfF4Jh
~/gerrit/git/RepoF.git/objects/pack/ MOVED_FROM tmp_pack_rG4dBh
~/gerrit/git/RepoF.git/objects/pack/ MOVED_TO .tmp-27710-pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ MOVED_FROM tmp_idx_NfF4Jh
~/gerrit/git/RepoF.git/objects/pack/ MOVED_TO .tmp-27710-pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ MOVED_FROM pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
---> ~/gerrit/git/RepoF.git/objects/pack/ MOVED_TO old-f4b8054434be1a227bdf8b12729efc1719f39708.pack   <----
~/gerrit/git/RepoF.git/objects/pack/ MOVED_FROM pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ MOVED_TO old-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ ATTRIB .tmp-27710-pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ MOVED_FROM .tmp-27710-pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ MOVED_TO pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ ATTRIB .tmp-27710-pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ MOVED_FROM .tmp-27710-pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ MOVED_TO pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
---> ~/gerrit/git/RepoF.git/objects/pack/ DELETE old-f4b8054434be1a227bdf8b12729efc1719f39708.pack  <----
~/gerrit/git/RepoF.git/objects/pack/ DELETE old-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ ACCESS pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ ACCESS pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.pack
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE pack-f4b8054434be1a227bdf8b12729efc1719f39708.idx
~/gerrit/git/RepoF.git/objects/pack/ OPEN,ISDIR 
~/gerrit/git/RepoF.git/objects/pack/ CLOSE_NOWRITE,CLOSE,ISDIR
Solution

  • If you do a standard git gc in a repository with no changes, this is expected. Git names its packfiles by a hash of their contents. Because Git doesn't recompute deltas for existing packs, when you git gc and there's only one pack and no loose objects, it's very likely that it will pack all the data into one pack that's the same as the old one.

    When this happens, Git still has a file descriptor to the old pack open because it doesn't close packs immediately. This is because often it's necessary to access them again, so it will try to leave them open a little while. The old pack, which is still open, is renamed to the old name, and the new pack is renamed into place; the old pack is then deleted. On a Unix system, it's completely possible to delete a file for which you have the file descriptor open; when the last process closes its file descriptor, the storage is freed.

    So this all seems completely normal for the scenario you're describing. Usually git gc is not a no-op, since additional objects are added to or removed from the pack or multiple packs are combined into one. But, if you do run a git gc immediately after running one with no intermediate changes, this is expected.