October 12, 2009
If a server says it’s low on disk space but you can’t find the offending file(s), try checking the output of lsof for “(deleted)”. Some services out there really really hate it when you poorly rotate logs and delete the old ones from under them.
Check free space:
[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 65G 52G 9.6G 85% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
Grasping at straws, check lsof:
[root@server ~]# lsof -p 2586 | grep "(deleted)"
lighttpd 2586 lighttpd 6w REG 8,1 33973933 7548243 /var/log/lighttpd/someotheraccesslog (deleted)<br />
/etc/init.d/lighttpd restart
Check fix:
[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 65G 16G 46G 26% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
All done!