On one of Linux development server that I use, I'm seeing 100% space usage at root "/" level. Root folder has a folder called "production" which is on a different mount / filer than the filer where root is. Root has close to 7.7/8 GB disk space.
I'm doing a search "du -chs" on all folder which are in MB or GB and also finding the total space used at Root "/" (excluding /production) but I'm not getting where the other space is and why it's 100% used when all folders (excluding /production) is showing that, it's only 2.8GB in total.
How can I get the missing space back?
Why df -kvh and du command doesn't reflect all the space getting used / not used correctly (when we say 100% used and du saying 2.8GB at Root "/" level)? Where is the approximate (7.7 allocated - 2.8 used) GB space?
Any idea! what I may be missing here.
[root@linux-server /]# df -kvh / && echo && echo && df -kvh /production
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-root
7.7G 7.3G 0 100% /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-production
258G 231G 14G 95% /production
[root@linux-server /]#
[root@linux-server /]# ls -1|grep -v "production" | xargs du -chs 2>/dev/null|egrep "total|[0-9][0-9]*[MG]"
7.8M bin
20M boot
106M etc
320M lib
25M lib64
60M root
39M sbin
1.9G usr
364M var
2.8G total
[root@linux-server /]#
OK. I don't know who down voted this question but it was important for me to find what was causing this behavior. I would like to share the answer with the community.
So, even though there was close to 5 GB free space at root "/", df command was not showing it, may be because some OS (bug / whatever by design).
It seems like if you kill a process and if it leaves some child process(es) behind OR if you kill a process which is using a folder and if you delete that folder (while the process is in progress), then this issue may occur.
Highlevel, if system does not somehow kill a child process/pid (which is not-required) process, your space won't be reflected correctly by "df" commmand (as that child/process still somehow holds the resources).
Now, how I resolved.
I saw there were some Jenkins (java) processes running, I killed them. That still didn't get my space back (I saw only 420KB free), then I did "ps -eAf" and saw the following process running.
c123456 23019 1 99 Oct21 ? 3-07:57:35 readelf -Wa /tmp/Goslr0qbOe/content2/jdk16-solaris-1.6.0.3.tar/jdk1.6.0_03/jre/lib/sparc/libioser12.so
(This above process was initiated by some utility, may be "pkgdiff" or some child step/process initiated by pkgdiff utility. One of my team member was comparing JDK 1.6 vs JDK 1.7 tar files and somehow this above process was still running even after I killed the main pkgdiff / related PIDs and also deleted /tmp/Goslr0qb0e two days back).
As soon as I killed the above PID (23019)... shenzi! I'm back in business.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-root
7.7G 2.5G 4.9G 34% /
Even after I restarted Jenkins instance(s), df now correctly shows it's 4.9G free for root "/" partition/mount/filer and that confirms that the above process was holding up the resources (which now I assume, is out of control/scope of "df" command's output).