Search code examples
shellinotifywait

Safe accessing inotified files

In a shell script I want to wait for a certain file to appear

inotifywait -m -r -q -e close_write -e moved_to --format '%w/%f' $ARCHIVE_ROOT | while read FILE; do
# Call other programs which process the file like stat or sha1sum
done

I would have assumed the file to be valid while in the handling code. Sadly the file seems to disappear e.g. after being processed by sha1sum. Did I miss something obvious, which is necessary to make the file last?

Solution

  • Many processes create temporary files and quickly delete them. Such is the nature of progress.

    To prevent a file from being deleted while your loop executes, use a hard link:

    inotifywait -m -r -q -e close_write -e moved_to --format '%w/%f' $ARCHIVE_ROOT | while read FILE; do
    ln "$FILE" tmpfile$$ || continue
    sha1sum tmpfile$$
    rm tmpfile$$
done

    If the file is destroyed before your while loop starts, there is nothing you can do. If, however, it exists at the start of the loop, the use of a hardlink will ensure that it continues to exist as long as you need it.

    The hard link with not stop other processes from changing the file. If that is a problem, then a copy (cp) is necessary.

    For this to work, the temporary file has to be on the same filesystem as the original.

    If this application is security-sensitive, you may want to use mktemp or other such utility to generate the name for the temporary file.