journalctl: A Practical Guide to the systemd Journal

It is 03:14 and a customer’s WordPress site is down. The PHP-FPM pool restarted four times in the last hour, then settled into a loop that php-fpm is silently swallowing. You have no access to a terminal where the app writes its own log, and the last line of text the site produced was a generic 502. What you do have is journalctl: the front end to systemd’s journal, which has been quietly collecting output from every unit on the box since the last boot — including the ones whose own logs are empty. The problem is that most admins know three of its flags and guess the rest under pressure, at 3am, with the customer refreshing the page. This guide is the set of invocations worth committing to muscle memory, plus the defaults that will bite you the moment you rely on them.

Reading the journal with the journalctl command

With no arguments, journalctl dumps the whole journal, oldest first, into your pager. That is almost never what you want — the useful work is in filtering. Three flags do most of the heavy lifting immediately:

journalctl -e          # jump straight to the end, in the pager
journalctl -n 200      # the last 200 lines, no pager
journalctl -n 50 -o cat # last 50 lines, message text only, no metadata

-o sets the output format. The default (short) prints one line per entry with a timestamp, hostname, syslog identifier and PID. cat strips everything but the message, which makes piping to grep far less noisy. When you need structure, -o verbose prints every journal field on its own line and -o json-pretty gives you the same as JSON — that is what you reach for when you want the _SYSTEMD_UNIT, _PID, or cgroup fields that the human formats hide.

-x adds explanatory text from the systemd message catalog beneath a message where one exists. For a unit that fails to start, journalctl -xeu php8.4-fpm.service frequently prints the catalog’s plain-English description of the exit code alongside the raw failure. It is the fastest way to turn status=226/NAMESPACE into something you can act on.

Filtering by unit and by time

For the 3am pager, unit plus time is the whole answer:

journalctl -u php8.4-fpm.service --since "1 hour ago"
journalctl -u nginx --since today --until "10:30"
journalctl -u ssh --since "2026-01-01 09:00:00" -p warning

-u matches a unit name (the .service suffix is optional). --since and --until accept the loose forms today, yesterday, 1 hour ago, -30min, or an explicit timestamp. Combine -u with another -u to watch several units at once.

One caveat on -u worth knowing before you trust it during an incident: it matches messages that carry the unit’s name in their metadata, which systemd derives primarily from the process’s cgroup. A daemon that deliberately escapes its cgroup — for example a script that re-parents a worker with setsid or a supervisor that forks a fully detached child — can leave the unit’s cgroup, and from that point its output is no longer tagged with the unit. It will still be in the journal, but -u thatunit will no longer show it. If a unit’s logs go suspiciously quiet, drop the -u and search by _PID or the syslog identifier instead.

Filtering by priority — and the trap it hides

This is the flag that catches out everyone who has not read the man page closely:

journalctl -p err

-p takes a priority name or number and shows that priority and everything more severe than it. It is a threshold, not an exact match. So -p err actually returns emerg, crit, alert and err messages together — everything at level 3 or worse. There is no built-in way to ask for “err messages only” with -p; if you truly need exactly one level you filter the JSON or the PRIORITY field yourself:

journalctl -p err --since today
journalctl -o json-pretty --since today | grep -A1 '"PRIORITY"'   # crude, for illustration

The valid names, from least to most severe, are debug, info, notice, warning, err, crit, alert, emerg. The numbers run 7 down to 0 in the same order. Because the comparison is a threshold, -p warning is the one you want when triaging: it drops the routine info chatter while keeping anything a service flagged as a problem.

Filtering by boot

The journal associates every entry with a boot ID, and that is how you survive a restart loop without drowning in old noise:

journalctl --list-boots          # boot IDs with time ranges
journalctl -b                    # this boot (default)
journalctl -b -1                 # the previous boot
journalctl -b -1 -u nginx        # nginx output from the previous boot

-b with no argument is the current boot; negative offsets walk backwards. --list-boots shows you how many boots the journal actually retains, which is the first thing to check when -b -1 comes back empty — see the next section for why that happens.

Following logs live

To watch a unit as it happens, -f follows the journal like tail -f. Seed it with recent context so you are not staring at a blank screen while you wait for the fault to recur:

journalctl -f -u php8.4-fpm.service -n 50
journalctl -f -p warning --since "5 minutes ago"
journalctl -f -u nginx -u php8.4-fpm.service

Everything you can filter by, you can follow: -f -u, -f -p, -f --since. Because journald already holds the history, you do not need to have been running -f before the incident to see it afterwards — that is the whole point of a central journal rather than an app-local file.

Why journald logs disappear after a reboot

Here is the default that surprises people the hardest. On a fresh Debian 12/13 or Ubuntu 22.04/24.04 install, journald keeps its journal in memory, under /run/log/journal — a tmpfs. It is wiped on every reboot. The relevant setting in /etc/systemd/journald.conf is Storage=, and its default value is auto:

$ grep -i '^#*Storage' /etc/systemd/journald.conf
#Storage=auto

Storage=auto means “write to /var/log/journal if that directory already exists, otherwise stay volatile in RAM.” It does not create the directory for you. So on a default box, journalctl -b -1 returns:

Specifying boot ID or boot offset has no effect, no persistent journal was found.

That error is the tell. To fix it, create the directory, let systemd set its ownership and mode, and restart journald:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Confirm it took effect:

journalctl --disk-usage
# ... should now report a path under /var/log/journal, not /run/log/journal

Two things to be precise about. First, anything already sitting in /run/log/journal stays there until the next reboot; the switch to persistent storage is not a backfill. Second, you can also set Storage=persistent explicitly, which tells journald to create the directory itself at boot — but the auto default already switches to disk-based journaling once the directory exists, so on an existing server the mkdir route is the minimal change and the one that matches how the man page describes auto.

Capping and vacuuming the journal

Once the journal is on disk, it grows. Check the size, then trim it on demand:

journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=2weeks
sudo journalctl --vacuum-files=5

--vacuum-size removes the oldest archived journal files until the total drops below the given size; --vacuum-time removes entries older than the given span; --vacuum-files keeps the newest N journal files. All three act on archived files and leave the currently-written file alone.

Vacuuming by hand is a stopgap. Set a cap in /etc/systemd/journald.conf so it never gets out of hand:

[Journal]
SystemMaxUse=1G
SystemMaxFileSize=100M
MaxRetentionSec=1month

Then sudo systemctl restart systemd-journald. SystemMaxUse caps the total disk footprint of /var/log/journal; RuntimeMaxUse does the same for the volatile /run journal. If you set neither, journald applies its own defaults, which scale with the size of the filesystem — which is why a large data volume can quietly accumulate gigabytes of journal before you notice. The exact default formula is version-dependent, so set explicit limits rather than relying on it. Remember that a manual --vacuum-* only takes effect now; the journald.conf ceiling is what keeps it bounded going forward.

journald and /var/log: they coexist

Enabling the journal does not replace the text files in /var/log, and on Debian and Ubuntu it does not even stop rsyslog from running. journald captures everything sent to the kernel log buffer, to the syslog socket, and to its own native socket, and by default also forwards those messages to syslog (the ForwardToSyslog setting). That is why a message shows up in both places:

journalctl -u ssh --since today | tail
sudo grep sshd /var/log/auth.log | tail

The two have different retention and both need attention. rotate /var/log with logrotate and cap the journal with SystemMaxUse as above; a disk-full alert that turns out to be the journal is a different fix from one that turns out to be syslog, and you want to be able to tell them apart at a glance. If you set ForwardToSyslog=no, the text files stop receiving journald’s copies — useful if you want the journal to be the single source of truth, but only once you are sure nothing else reads those files (fail2ban, log-shipping agents and monitoring can all depend on them).

The short version

1. Filter, don’t dump: -u unit, --since/--until, -p warning, -b, -n, -x.
2. -p is a threshold — -p err shows err and everything more severe, not err alone.
3. -u follows the cgroup; a process that escapes it stops being tagged with the unit.
4. Default Storage=auto stays in RAM unless /var/log/journal already exists — that is where -b -1 turns up empty.
5. Make it persistent with mkdir -p /var/log/journal, systemd-tmpfiles --create --prefix /var/log/journal, then restart journald. No backfill.
6. Cap it in journald.conf with SystemMaxUse; trim now with --vacuum-size/--vacuum-time.
7. journald forwards to syslog by default, so /var/log and the journal grow separately — cap both.