Finding deleted files that are still opened by a process

Recently my server ran out of space, but my checks (via du) reported that only about 60GB was actually being used. That's because if a process has a file open that is then deleted, the space is still claimed, but the file doesn't appear.

The following code snippet will show files that have been deleted, but are still being held open (and using storage space).

lsof +L1

That's it. From there, you can determine what applications were still holding open files. In my case, rsyslog was still holding open an old version of /var/log/syslog.1 and /var/log/messages.1:

rsyslogd  30120       root    1w   REG    8,3 25107334316     0   1211404 /var/log/syslog.1 (deleted)
rsyslogd  30120       root    2w   REG    8,3 18718398817     0   1211409 /var/log/messages.1 (deleted)

You'll notice that they show deleted, and have a combined storage footprint of approximately 43GB. Restarting rsyslog allowed the files to be freed, and storage reclaimed.