Search code examples
loggingdisktailsunos

"tail -f" makes disk full?

our application server (sunOS) always gets disk full. and our Infrastructure team said it's caused by too many "tail -f" processes. Because the app rotates log file frequently, it caused the dead link and no disk space? I've never heard of this before. does that command really cause disk full?

Solution

  • The space a file occupies cannot be reclaimed until all references to that file are gone. Therefore, any process that has the file open will prevent the file from being deleted from the disk.

    An active tail -f following the file, for example.

    If these files need to be deleted to free disk space (e.g. because they are very big, or there are very many of them), then having processes lying around that hold references to them will prevent their deletion, and eventually lead to the disk filling up.

    Edit in response to the comment on the other answer:

    The diagnostics you report are exactly what you would expect to see in the situation that Adam and I describe. df reports that 56G of disk are in use, and du reports that only 10G are visible in the folder. The discrepancy is because there are 46G worth of files that have been removed from the folder, but cannot be physically removed from disk because some processes are holding a references to them.

    It's easy enough to experiment with this yourself: find a filesystem it's safe to play with, and create a humongous file. Write a C program that opens the file and goes into an infinite loop. Now, do the following:

    • Start the program
    • Check the output of df
    • rm the file
    • Check the output of df again
    • Stop your program
    • Check the output of df again

    You will see that the output of df doesn't change after rming the file, but does change once you stop the program (thus removing the last reference to the file).

    If you need even more evidence that this is what's going on, you may be able to get information from the /proc filesystem, if you have it. Specifically, find the PID of one of the tail -f processes (or other processes you think might be the cause), and look at the directory /proc/<pid>/fd to see all of the files it has open.

    (I don't have *nix at home, so I can't actually check to see just what you'll see /proc/<pid>/fd in this situation)