Where Did My Disk Space Go? Finding and Freeing Space on a Full Linux Server

The 3 a.m. message every sysadmin knows

Your phone buzzes. A client’s site is down, or a cron job just started failing with “No space left on device.” You SSH in, run df -h, and there it is:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        79G   78G     0 100% /

Now what? The instinct is to start deleting things. Don’t. The first job is to find out where the space actually went, because a full disk is almost never caused by what you think. It’s rarely the website. It’s usually a log file, a runaway backup, or a deleted-but-still-open file that no du command will ever show you.

Here is the order I work in. It goes from fast and broad to slow and specific, and it will find the culprit on almost any Linux server.

Step 1: Confirm the filesystem, not just the disk

Most servers have more than one mount point, and “the disk is full” usually means one specific filesystem is full. Always check the whole picture before touching anything:

df -h

Look at the Mounted on column carefully. A server with a full /boot behaves completely differently from one with a full /. If you see a separate /var or /home mount, that narrows your search enormously — you already know which tree to hunt in.

Two flags worth knowing:

• df -h --total adds a summary line at the bottom.
• df -i shows inode usage instead of block usage. A disk can be 40% free by bytes and still refuse to write files because it’s out of inodes. If df -i shows 100% on IUse%, jump straight to Step 7.

Step 2: Find the fat directories with du

This is the workhorse. Start at / and go one level deep, sorted biggest first:

du -x -h -d 1 / 2>/dev/null | sort -rh | head -20

That’s doing three important things at once:

• -x stays on one filesystem, so you don’t waste time walking into mounted network shares or other disks.
• -d 1 limits output to one directory level deep.
• 2>/dev/null silences the flood of “Permission denied” noise you’d otherwise get on /proc and friends.

Read the result, then drill into whichever directory is largest:

du -x -h -d 1 /var 2>/dev/null | sort -rh | head -20

Repeat, narrowing each time. Three or four iterations usually lands you on the exact directory. The usual suspects on a web server are /var/log, /var/lib/mysql, /home/*/logs, /var/lib/docker, and /var/cache.

Step 3: Find individual large files

Sometimes it’s not a directory of many files — it’s one enormous file. This finds anything over 1GB, skipping virtual and other filesystems:

find / -xdev -type f -size +1G -exec ls -lh {} + 2>/dev/null | sort -k5 -rh | head -20

Change +1G to +500M if nothing turns up. Common finds here: a giant mysql-slow.log, an orphaned .tar.gz someone left in /tmp, a core dump in the application directory, or a debug log a developer forgot to disable.

Step 4: ncdu, when you want to browse

If you’d rather scroll than type, install ncdu. It walks the tree once, then gives you an interactive interface where you can arrow into directories and see sizes update live:

apt install ncdu
ncdu -x /

Press d to delete a highlighted file or directory without leaving the tool. It’s the fastest way to explore an unfamiliar server, and it’s genuinely pleasant to use.

One warning: ncdu and du both scan the whole tree, which can take minutes on a busy server and will hammer the disk doing it. On a production box under load, prefer targeted du commands over a full-tree scan during peak hours.

Step 5: The one that fools everybody — deleted files still held open

This is the case that makes experienced people doubt their tools. You free up 20GB, you run df -h, and the usage hasn’t moved. Where did the space go?

On Linux, when a process deletes a file, the space is only reclaimed once nothing has it open. If a long-running process — a PHP-FPM worker, a database, a logging daemon — still holds a handle, the file is invisible to du and find but still occupying real blocks on disk.

Find them like this:

lsof -nP +L1 2>/dev/null | head -30

The +L1 filter means “list files with a link count lower than 1” — precisely the deleted-but-open case. The output shows the process, the PID, and the size it’s still holding. It’s not unusual to find a single rotated log at 30GB that no filesystem tool will admit exists.

The fix is to get the process to release the handle. Restarting that service reclaims the space immediately:

systemctl restart php8.4-fpm

Restart the specific process responsible, not the whole server. And if this is happening to you regularly, it means something is rotating logs without the daemon being told to reopen them — see Step 6.

Step 6: Logs, the number one cause on web servers

On a hosting box, logs are almost always the answer. Two places to check first:

du -xh /var/log/* 2>/dev/null | sort -rh | head -15
journalctl --disk-usage

If systemd’s journal is the problem — and it often is, because it’s uncapped by default — trim it and put a hard limit on it permanently:

journalctl --vacuum-size=500M
# then make it permanent:
sed -i 's/^#SystemMaxUse=.*/SystemMaxUse=500M/' /etc/systemd/journald.conf
systemctl restart systemd-journald

For classic text logs, check that logrotate is actually running and actually configured for the files that are growing. A surprising number of servers have a 40GB access.log that was never added to a rotation config:

logrotate -d /etc/logrotate.conf      # dry run: what would it do?
cat /var/lib/logrotate/status         # when did it last run?

If a daemon keeps writing to a file after logrotate moves it, you’ll hit Step 5’s deleted-open-file trap. Most daemons handle SIGHUP correctly, but some packaged applications don’t — that’s when you switch to copytruncate in the logrotate stanza.

Step 7: When it’s inodes, not bytes

If df -h shows plenty of free space but writes still fail, check inodes:

df -i

An inode is consumed by every file, directory, and symlink. Millions of tiny files will exhaust them while barely denting your disk usage. The classic culprits are PHP session files, mail queues with thousands of stuck messages, and enormous cache directories.

# count files per directory, deepest offenders first
for d in /var/*; do printf "%8s  %s\n" "$(find "$d" -xdev 2>/dev/null | wc -l)" "$d"; done | sort -rn | head

Once you’ve found the directory, deleting the stale files reclaims inodes. If this is structural — a mail queue with a spam backlog, for example — fix the upstream cause or it will come straight back.

Step 8: Caches, containers and old kernels

Three more places that quietly accumulate gigabytes:

# package manager cache
apt-get clean

# old kernels (keep the running one and one spare)
apt-get autoremove --purge

# Docker: unused images, containers, volumes, build cache
docker system df
docker system prune -a --volumes

The docker system prune -a --volumes command is destructive — it removes all unused images and volumes, not just dangling ones. Read docker system df first and make sure you understand what will go.

Step 9: Make sure it doesn’t happen again

Freeing space is the firefighting. You want the monitoring. Three cheap habits that prevent the 3 a.m. call entirely:

• Alert on disk usage, not on downtime. A threshold alert at 80% gives you days of warning. Most monitoring systems treat a full disk as a symptom; you want it as a trigger.
• Watch inodes too. Almost nobody monitors inode exhaustion, which is why it catches people out. Your monitoring tool can graph both.
• Cap your logs at the source. Set SystemMaxUse for the journal, and review your logrotate configs whenever you deploy something new that writes logs.

The short version

When a disk fills up, work in this order:

1. df -h — which filesystem, and is it bytes or inodes?
2. du -x -h -d 1 — walk down the biggest directories.
3. find / -xdev -size +1G — any single monster files?
4. lsof -nP +L1 — deleted files held open by running processes. This is the one people miss.
5. Logs, journald, package cache, Docker, old kernels.
6. Set alerts at 80% and cap your logs so it never happens again.

None of this requires special tooling. It requires knowing the order — and knowing that df and du can disagree for a perfectly good reason.