Diagnosing High Load Average on Linux

The alert fires at 03:12: load average is 47 on a 32-core database host. You SSH in expecting a fire, and instead find top showing 84% idle CPU, a warm page cache, and query latency that never moved. Nothing is actually slow. So what is that 47? If you have ever paged a DBA over a number that turned out to be meaningless, you have met the most misunderstood metric in Linux. This is a practical guide to diagnosing a linux high load average without panicking at a number you do not understand.

Linux high load average: what the number actually counts

The load average is not a CPU percentage, and it never has been on Linux. It is the number of tasks that are either runnable (state R — waiting for or using a CPU) or uninterruptible (state D — blocked in the kernel, almost always on I/O) at a given instant, averaged over 1, 5, and 15 minutes. That is it. It is a count of tasks, not a measure of how busy your CPU is.

You can read the raw values straight from the kernel:

$ cat /proc/loadavg
47.12 38.90 21.44 12/1841 88213

The first three fields are the 1, 5, and 15 minute averages. The fourth is runnable/total tasks right now (12/1841 means 12 runnable out of 1841 total). The last field is the most recently allocated PID. uptime prints the same averages with a friendlier wrapper.

Two details matter more than most articles admit. First, this is the number of tasks, and since the 2.6 kernel a task equals a thread — a multithreaded process contributes one unit per active thread, not one per process. Second, the kernel does not measure this continuously. It samples the run and blocked queues roughly every five seconds and decays the running averages exponentially, so the “1-minute” figure is a smoothed estimate, not a precise window. Treat all three numbers as trends.

Why a database box can sit at load 40 and be perfectly healthy

Back to our 47. The database is doing heavy buffered reads and writes. Its worker threads spend most of their time in state D, parked in the kernel waiting for storage or a filesystem lock to return. Each of those tasks adds 1.0 to the load average even though not one of them is competing for a CPU core. A workload dominated by D-state tasks can push load average arbitrarily high while the CPUs sit idle — the classic high-load/low-CPU signature of an I/O-bound box.

This is the first thing to internalise: on Linux, a D-state task is counted the same as a R-state task. Historically the BSD-derived Unices counted only runnable tasks; Linux chose to include uninterruptible sleep so the metric would reflect storage stalls too. The cost is that load average conflates two very different problems — not enough CPU, and waiting on slow I/O — into a single figure.

Load is not CPU saturation

Because the two are conflated, load average and CPU saturation can diverge in both directions:

• High load, low CPU: I/O-bound work, D-state tasks, NFS hangs. Load climbs, cores idle.
• Low load, high CPU: a single-threaded process pinning one core on a 32-core box shows load around 1.0 — but if it is your latency-critical service and it is 100% busy on that one thread, you have a real problem the load average will never surface.

Load average also cannot tell you which of those your host is suffering. That is why you never diagnose from uptime alone. You have to look at the composition of the queue.

Breaking it down: uptime, top, vmstat, pidstat

Each tool answers a different question. Use them together:

• uptime / /proc/loadavg — the headline number and its trend.
• top — sort by %CPU for CPU hogs, but more importantly read the task-state line and the %wa column to see how much of the load is waiting on I/O.
• vmstat 1 — the run queue, the blocked queue, context switches, and the CPU breakdown, one line per second.
• pidstat — attribution: which process, which thread, CPU or disk.

Start with the composition of the queue rather than the top of the process list:

$ ps -eo state,pid,ppid,wchan:32,comm | awk '$1=="D"'
D    3141      1 rpc_wait_bit_killable      nfs_worker
D    9017   8920 xfs_log_force              postgres
D    9018   8920 xfs_log_force              postgres

The wchan column names the kernel function each task is sleeping in — here, NFS waits and XFS log flushes. That single line often solves a “load 40” mystery in seconds. Note that a task in D cannot be killed; kill -9 will not interrupt it, and it will not disappear until its I/O returns or times out.

Run queue versus blocked queue: reading vmstat’s r and b

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  2      0 812340  92100 4021188    0    0  1024  8800  311  902 12  4 78  5  1
16  1      0 809912  92100 4022412    0    0  2048  9100  330  940 13  4 77  5  1
15  3      0 810004  92100 4023010    0    0   512  8600  298  880 12  4 79  4  1

Two columns are the whole game. r is the number of runnable tasks waiting for a CPU slice — the run queue. b is the count blocked in uninterruptible sleep, the contributors to that I/O-heavy load. In this sample, r stays around 14–16 while core count matters: keep reading. The CPU columns at the right (us, sy, id, wa, st) tell you where the cycles went, and wa shows the I/O-wait fraction.

A healthy I/O-bound box typically looks like low r, elevated b, and non-trivial wa. A CPU-contended box looks like sustained r greater than your core count, near-zero wa, and high us+sy. Those are opposite diagnoses hiding inside the same headline load number.

I/O wait versus CPU contention

%wa is the fraction of time CPUs were idle because a task was waiting on disk. It is the tool that separates “the storage is slow” from “we do not have enough CPU.” Add iostat -x 1 from the sysstat package to confirm: a device pegged near 100% %util with high await explains an I/O-driven load spike, while a quiet disk with a saturated run queue points squarely at CPU.

Here is a detail that fools a lot of people: iowait is attributed per-CPU and then averaged across all of them. On a 64-core host with one genuinely bottlenecked disk, the aggregate wa in vmstat or top can read as 1–2% because the other 63 cores were simply idle, not waiting. The I/O problem is real; the percentage just got diluted. Do not dismiss an I/O wait of a few percent on a large machine without checking the per-device numbers.

Why the core count changes everything

A run queue of 14 means completely different things on 4 cores and 64. You normalise by dividing:

$ nproc
32
$ uptime
 03:14:07 up 41 days,  6:22,  load average: 47.12, 38.90, 21.44

Load 47 on 32 cores is roughly 1.5 tasks per core — mild contention, not a disaster, and entirely consistent with a busy database. The same 47 on a 4-core VPS would be a five-alarm fire. Load average is only meaningful once you know the CPU count, which is why “load should be under 1” is folk wisdom that only applies to a single-core machine. Use nproc or getconf _NPROCESSORS_ONLN. On hosts with SMT you may want physical cores rather than threads for a strict read; lscpu breaks that down.

Containers and virtualisation: the numbers lie

This is where load average misleads most often, and it is the trap most write-ups skip entirely: the load average is not namespaced. Containers have their own PID, mount, network, and UTS namespaces, but there is no load namespace. /proc/loadavg and uptime inside a container report the host’s load, because the container shares the host kernel that computes it. A cron container doing nothing can show load 30 purely because a noisy neighbour on the same node is thrashing its disk. If you run containers, never alert on load average read from inside one — it is someone else’s number.

Worse, the container’s actual limit is invisible to load. A cgroup v2 CPU quota of one core means your processes can be throttled at 100% of a single CPU while host load sits near zero. Check the quota and throttle counter directly:

$ cat /sys/fs/cgroup/cpu.max
100000 100000
$ grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/cpu.stat

100000 100000 means 100ms of runtime per 100ms period — one CPU. Rising throttled_usec while CPU is idle tells you the quota, not the machine, is your ceiling.

On virtual machines, watch the st column (steal time) in vmstat, or the steal field in /proc/stat. Steal is time your vCPU was runnable but the hypervisor gave the physical core to someone else. High steal means the host is oversubscribed and your “load” is partly somebody else’s workload leaking into your accounting.

If your kernel is 4.20 or newer, PSI (pressure stall information) is a far better signal than load average and it is cgroup-aware. It reports the fraction of time tasks were actually stalled on CPU, memory, or I/O:

$ cat /proc/pressure/cpu
some avg10=0.14 avg60=0.09 avg300=0.05 total=812345
$ cat /proc/pressure/io
some avg10=6.20 avg60=4.10 avg300=2.30 total=998877
full avg10=1.90 avg60=1.20 avg300=0.70 total=301244

For I/O, full is the number to watch: it is the time when all tasks were stalled, meaning nothing made progress. A load average of 40 with flat PSI is a machine doing useful buffered I/O. A modest load with climbing full pressure is a machine that has genuinely stopped. Version note: PSI requires a kernel built with CONFIG_PSI and, until recently, the psi=1 boot parameter on some distributions — check whether /proc/pressure/ exists before relying on it.

The short version

• Load average counts runnable (R) plus uninterruptible (D) tasks, averaged over 1/5/15 minutes. It is a task count, not a CPU percentage.
• D-state tasks — usually I/O waits, NFS hangs, or filesystem flushes — inflate load while CPUs sit idle. A database box at load 40 with idle cores is normal.
• Normalise by core count. Load 47 is fine on 32 cores and dire on 4.
• Break it down: ps ... awk '$1=="D"' for blocked tasks and their wchan; vmstat 1 for the run queue r, blocked queue b, and wa; pidstat -d and pidstat -u to attribute to a process; iostat -x to confirm the disk.
• Distinguish I/O wait from CPU contention — they are opposite fixes and hide behind the same load number. Remember aggregate wa is diluted across cores.
• Load average is not namespaced: containers report the host’s load, so never alert on it from inside one. Use the cgroup CPU quota, the throttle counter, and PSI instead.