A Practical Guide to rsync for Server Backups

The call comes in at 09:40: a customer deleted a directory of uploads two weeks ago and only noticed now. You open the backup server, an ageing box with a 2 TB disk that runs a nightly rsync -a --delete /srv/www/ /backup/www/ from cron. The job has been reporting success every night for eight months. You look for the deleted directory and it is not there — it was gone the first night after the deletion, and every night since. This is the single most common way a working sysadmin ends up with a rsync backup server that does not actually back anything up. What follows is the detail that separates a real backup from a sync that deletes your history for you.

On an rsync backup server, a mirror is not a backup

rsync with --delete produces a mirror: an exact copy of the current state. That is genuinely useful for failover and for keeping a standby in lockstep, but it is not a backup. There is no history. Whatever happens on the source — a fat-fingered rm, a botched deploy, ransomware that encrypts everything in place — is faithfully reproduced on the destination on the next run.

A backup needs point-in-time history: you must be able to ask “what did this directory look like on the 3rd of last month?” and get an answer. The two designs are not interchangeable, and if you are reaching for --delete on a nightly cron, decide first which one you are building.

The trailing slash changes the meaning of the command

This trips people up for years. On the source, a trailing slash means “copy the contents of this directory”. Without the slash it means “copy the directory itself”. Both commands below are correct; they produce different trees.

$ rsync -a src/ dest1/
$ find dest1
dest1
dest1/file.txt
dest1/sub
dest1/sub/nested.txt

$ rsync -a src dest2/
$ find dest2
dest2
dest2/src
dest2/src/file.txt
dest2/src/sub
dest2/src/nested.txt

So rsync -a /srv/www/ /backup/www/ puts file.txt at /backup/www/file.txt, whereas rsync -a /srv/www /backup/www puts it at /backup/www/www/file.txt. The trailing slash on the destination does not matter — only the source matters. Pick a convention (I always use a trailing slash on both when mirroring contents) and enforce it everywhere, because mixing the two across a script-or-hand-run boundary is exactly how you get a double-nested tree nobody notices for a month.

The –delete flag and the direction that bites

--delete removes files on the receiving side that are not present on the sending side. Get the argument order wrong — destination first — and rsync will happily treat your live data as the thing to be pruned to match the backup. rsync does not stop you. There is no confirmation prompt.

The subtler failure is not a swapped argument but a source that is temporarily empty. If /srv/www is an NFS or CIFS mount that has failed, or a directory that only exists after another service starts, rsync sees an empty source and, with --delete, does exactly what you asked:

$ ls dest
customer_data.tar
$ rsync -a --delete src/ dest/    # src is empty because the mount is missing
$ ls dest
$                                 # backup wiped, exit code 0

Guard against this in the wrapper script. Refuse to run the delete pass unless the source mount is present and non-trivial in size. A cheap check is a sentinel file you know must exist:

mountpoint -q /srv/www || { echo "source not mounted" >&2; exit 1; }
test -e /srv/www/.backup-ok || { echo "sentinel missing" >&2; exit 1; }

For the hammer-shy, dry-run first: rsync -a --delete --dry-run -i src/ dest/ lists precisely what would be transferred and deleted, and touches nothing.

Excluding caches and sockets — and the trap that leaves them behind

On a live server you never want to copy .cache, node_modules/.cache, session stores, /tmp scratch, or Unix sockets. The straightforward exclusion is a filter list:

rsync -a --delete \
  --exclude='*.sock' \
  --exclude='.cache/' \
  --exclude='*/.cache/' \
  src/ dest/

Sockets deserve a specific callout because -a does something surprising. The archive flag implies -D, which includes --specials, so rsync will try to recreate Unix domain sockets on the destination. Here is the proof, and the fix:

$ rsync -av s/ d/
sending incremental file list
app.sock
$ ls -l d/
srwxrwxr-x 1 liam liam 0 app.sock        # a socket, faithfully recreated

$ rsync -av --no-specials s/ d2/
sending incremental file list
skipping non-regular file "app.sock"

Copying a socket is meaningless — the file is a rendezvous point for a running process, not data — and it can confuse restore tooling. Add --no-specials whenever you back up anything that might contain /run, PHP-FPM pools, or systemd sockets.

Now the non-obvious trap. Excluded files are invisible to --delete. A file that matches an exclude rule is never considered for deletion on the receiving side, even when it is absent from the source, because rsync has been told to ignore it entirely. Watch:

$ ls d
skip.bak
$ rsync -a --exclude='*.bak' --delete s/ d/
$ ls d
keep.txt
skip.bak          # still there — never pruned
$ rsync -a --exclude='*.bak' --delete-excluded s/ d/
$ ls d
keep.txt          # now gone

So a mirror with broad excludes is quietly not a clean replica: stale junk matching your filters accumulates forever. If you want the destination to be pruned to match the source including the excluded set, you must add --delete-excluded — and understand that it does delete previously-excluded files. Most people want plain --delete plus the excludes; just know which one you have.

Real incrementals with hard-link snapshots

When you do want history without copying every byte every night, --link-dest gives you space-efficient snapshots. Each run writes a new directory, but files unchanged since the reference run are hard-linked instead of copied. A hundred snapshots of a 500 GB tree can cost little more than one copy plus the deltas. Verified inode behaviour:

$ rsync -a a/ "$D/c" --link-dest="$D/a"
$ ls -li a/one.txt c/one.txt
410974 -rw-rw-r-- 2 ... a/one.txt
410974 -rw-rw-r-- 2 ... c/one.txt      # same inode: hard-linked

The link count 2 is the giveaway. The standard rotation looks like this:

DEST=/backup/www
D=$(date +%F)
rsync -a --delete --link-dest="$DEST/current" "$SRC/" "$DEST/$D/"
rm -f "$DEST/current"
ln -s "$D" "$DEST/current"

Two rules keep this honest. First, --link-dest must be a relative path or an absolute path into the same destination tree, and every snapshot must live on the same filesystem — hard links cannot span filesystems, so putting older snapshots on a mounted sub-directory silently turns them into full copies and fills the disk. Second, never run --delete against an old snapshot directory as a source; prune whole snapshots with rm -rf on the dated directory, not file-level surgery.

Pushing it over SSH

Remote transport is just a remote spec with a colon. Use it to pull, so the backup server reaches out and the web server never holds write access to the backups:

rsync -a --delete --no-specials \
  -e 'ssh -p 2222 -o StrictHostKeyChecking=yes' \
  www-host:/srv/www/ /backup/www/

Use a dedicated key restricted in ~/.ssh/authorized_keys with a command= forced-command wrapper or, on the sending side, rrsync (the --server helper shipped with rsync) to lock the key to one directory tree. On the receiving host, run the daemon behind --server rather than exposing port 873: SSH gives you encryption and authentication for free. Add -z only if CPU is cheaper than bandwidth on your link — on a fast LAN it is pure cost.

Resume behaviour when the link drops

By default rsync deletes an interrupted file and re-sends it whole next time. For a 200 GB database dump over a flaky link that is useless. Three flags matter:

• --partial keeps the partially-transferred file so the next run continues from where it stopped.
• --partial-dir=.rsync-partial parks partial files in a subdirectory, so a torn file never sits at the real path where a service might try to read it.
• --append-verify resumes by checksumming the existing data on the receiver and appending only the remainder.

The caveat: --append and --append-verify assume the file has only ever grown and that the leading bytes are identical on both ends. That holds for append-only logs and database dumps and not much else. Point them at a file that may have been rewritten in place and you will silently splice mismatched data together. rsync’s own manual warns they are dangerous unless you are certain every file in the transfer is a shared, growing file — so use them for a specific --include/--exclude set, not across the board.

Verify the restore, not just the copy

An rsync job exiting 0 proves the transfer ran, not that the backup is usable. A backup nobody has restored is a hypothesis. Two things to build into the routine:

First, understand the exit codes, because they are not all failures. 0 is success, 23 is a partial transfer due to error, 24 is a partial transfer because source files vanished during the transfer, and 25 means --max-delete stopped deletions. Exit 24 is normal on a busy server — files churn mid-run — so a wrapper that alarms on any non-zero exit will cry wolf nightly, while a wrapper that ignores non-zero entirely will miss real 23 errors. Handle 24 as benign and escalate everything else.

Second, periodically restore. Copy a sample tree to a scratch path and compare checksums, not timestamps:

rsync -a --checksum -n -i /backup/www/ /srv/www-verify/ \
  | grep -v '^\.d' | head

--checksum forces a content comparison rather than the default size-and-mtime heuristic, and -n (dry-run) plus -i (itemise) prints any file that differs — an empty list means the backup matches the source byte-for-byte. Do it on a schedule, and actually extract a file or a database dump from the snapshot and open it. The morning you need it is the wrong time to discover the restore path was never tested.

The short version

• A --delete mirror has no history. Decide up front whether you want a standby mirror or a point-in-time backup; they are different tools.
• A trailing slash on the source copies the contents; without it you get a nested copy of the directory itself. Only the source’s slash matters.
• Never run --delete without guarding against an empty or unmounted source — that is how backups get wiped with exit code 0.
• Excluded files are invisible to --delete; add --delete-excluded only if you truly want them pruned.
• -a recreates Unix sockets — add --no-specials. Exclude caches and scratch dirs explicitly.
• Hard-link snapshots via --link-dest give cheap incrementals, but keep every snapshot on one filesystem or they silently become full copies.
• Pull over SSH with a restricted key; use --partial-dir and --append-verify only for genuine append-only growing files.
• Treat exit 24 as normal, alarm on 23, and prove the backup with a real checksum-compared restore — not with a green cron mail.