Steve Siegfried wrote:

Dan Track wrote:

...

Hi

I've got a server with 59G /opt partiition. Currently df -h reports that I'm using 59G, however if I run "du -hs" on /opt the size of the indidivaual dir's sitting in /opt are much smaller and don't sum up to 59G. So my question is what is making the server think that more disk space is being used?

Any help would be appreciated.

It might be useful if you showed the du & df output, and that you should remember to run du as root in order to see all the directories and not just the ones you have permission to read as a normal user.

Suggest also that you read the fedora archives for the topic "df cmd calculates wrong percent usage (FC6)".

-S

Also remember that space is not freed when a rm command is issued. It is freed when the file is afterwards closed.

It is quite possible to have files open that do not show in the directory listing.

Imagine a large file such as a database file that is being used by a mysql process. Lets also assume that this mysql process is involved in heavy access to that very file doing a report that takes hours.

It is possible to rm that file out from under the process. However, as long as that process holds the file open, nothing has happened except the inode has been removed from the directory listing. At that point, df does not show any more space available than before the rm command. That is because the kernel driver for the file system does not need to free up the space until the file is closed.

Different types of file systems behave differently, but most of them will behave this way.

Never trust df on a busy file system except at boot up. :)

Good luck!

Hi Dan,

Dan Track wrote:

On 2/27/07, Steve Siegfried sos@zjod.net wrote:

[snip]

...

Hi

Thanks everyone for your replies. Here's the relevant output. Is there some tests I can run to see what is going on?

df -h Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p3 3.0G 1.9G 967M 67% / /dev/cciss/c0d0p1 147M 15M 125M 11% /boot /dev/cciss/c0d0p7 59G 53G 3.3G 95% /opt none 1007M 0 1007M 0% /dev/shm /dev/cciss/c0d0p2 3.0G 2.5G 358M 88% /var

du -hs /opt/ 27G /opt

I've seen this before on a suse box where the user had couple downloads running that he canceled but the browser didn't let go of the files. His filesystem was filled 100%. After he killed his browser df -hl showed the correct information. Maybe you can see what's going on by running

/usr/sbin/lsof /opt | less

on your box.

Alex

This topic was handled off-list, which was probably a mistake. Thus, for you edification and enjoyment....

Basically, files can temporarily "disappear" from the file-system when a program: - opens a file with fopen/open/etc, and then - unlinks the file (i.e.: removes the file, see unlink(2)) while keeping the the file open.

Note that when this happens, the disk space isn't really available for reallocation until the program ends and/or closes the file (fclose/close/etc). However, because of the way df and du calculate used and free file-system space, they can disagree under these conditions.

Also note that USED + AVAIL != SIZE for df because some of the space on a disk file-system is reserved for use by the system (for superblocks, "reserved", etc).

So how did you get here? - downloads that got canceled - some program that's messed up and poops all over its log or output file (that's been unlinked but is still open) - a log rotation that went awry - processes listed as "zombie" in ps/top output - a make where the user hit cntl-C (unlikely, make's pretty good about this kind of thing) - a daemon or long running program that poorly manages files - etc

If you can find and zap the PID that's tying up the disk space, you'll probably see the disk space reappear. If not, you can reboot the server. I'd suggest rebooting and then repeating the df/du immediately afterward.

The quickest way to find PIDs that have deleted (i.e.: NLINK == 0) but still open files is $ su # lsof +L 1 /opt | grep deleted

BTW: lsof is in the lsof rpm.

If this _still_ doesn't solve the issue (to say within 5%), then you'll need to unmount the drive and run "fsck -f /dev/cciss/c0d0p7". Note that while fsck will fix damaged filesystems, it won't find/fix files that lsof reports as deleted but still open.

-S

shhgs wrote:

Hi, Dan

Maybe someone has removed a file while a process is still writing to it. Try fcsk.

G.S. Huang

On 2/28/07, Alexander Apprich a.apprich@science-computing.de wrote:

...

Hi Dan,

Dan Track wrote:

...

On 2/27/07, Steve Siegfried sos@zjod.net wrote:

[snip]

...

...

Hi

Thanks everyone for your replies. Here's the relevant output. Is there some tests I can run to see what is going on?

df -h Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p3 3.0G 1.9G 967M 67% / /dev/cciss/c0d0p1 147M 15M 125M 11% /boot /dev/cciss/c0d0p7 59G 53G 3.3G 95% /opt none 1007M 0 1007M 0% /dev/shm /dev/cciss/c0d0p2 3.0G 2.5G 358M 88% /var

du -hs /opt/ 27G /opt

I've seen this before on a suse box where the user had couple downloads running that he canceled but the browser didn't let go of the files. His filesystem was filled 100%. After he killed his browser df -hl showed the correct information. Maybe you can see what's going on by running

/usr/sbin/lsof /opt | less

on your box.

Alex