Backups

Dump databases inside their own containers, back up every stack's data with restic and prove both restore.

Commandsdb lsdb backupdb restore-testbackup setupbackup runbackup lsbackup imagebackup restore-testbackup restorebackup resumevolumes clean-aside

Dump and restore test

seawall db ls

output · example data

> web-db mariadb:12.3.3 (web)
  ╰ no dumps yet: seawall db backup ...
> blog-db mariadb:12.3.3 (blog)
  ╰ latest 5h ago, 0.7 MiB, not restore-tested  /home/admin/backups/blog/20261022T124151Z-blog-db.sql.gz
seawall db backup nextcloud --yes
seawall db restore-test nextcloud --yes

db backup dumps a MySQL, MariaDB or Postgres database inside its own container to ~/backups/<stack>/. Dumps are 0600 in 0700 folders. db restore-test loads the latest dump into a throwaway container with no network and compares table counts. The nightly timer at 04:30 UTC (timers.db_backup, in timers.zone) dumps and tests every database and keeps 14. stack save keeps only the deployed compose file. The data backup below takes the volumes and folders.

Postgres is dumped whole with pg_dumpall. The password comes from the container's own POSTGRES_PASSWORD and reaches the client through PGPASSWORD, never its arguments. The restore test reaches each database through PGDATABASE, so a database whose name reads like a connection string is still only a name. It compares the tables in every database with the live one.

Data backup

seawall backup setup --to /mnt/backup/seawall
seawall backup setup --to /mnt/backup/seawall --yes
seawall backup run
seawall backup ls

backup setup makes restic's repository in a folder for root alone and a password for it. --password-env NAME takes the password from that variable instead. Setup also installs seawall-data-backup.timer, which runs every day at 04:45 UTC (timers.data_backup), after the database dumps.

Warning

The password is shown once and kept 0600 beside seawall.toml. Keep a copy away from the host, as the backup can't be read without it.

backup run takes each stack's volumes and its folders under the stacks folder, plus ~/backups, one snapshot each. Without --yes it lists what it would read and what it leaves out and why. A database's data volume is left out only while a verified dump from the last day covers it. restic runs from an image seawall builds from restic's source on a pinned Go: The source, Go and the built binary are each checked against a pinned sha256, and restic's modules against its go.sum. It runs with no network for a local folder, a read-only root and only the capability to read.

backup image says whether that image is built on each runtime the backups use, and --yes builds what's missing. A failed build names the step that failed and its last lines, and exits 2.

Note

Run backup image after you install or upgrade seawall, so the night's run has nothing to fetch.

Each stack is read as [backup.stacks] in seawall.toml says. pause is the default and freezes its containers while restic reads. live reads it running, stop stops it and starts it again with its container alerts silenced, and exclude leaves it out. A stack holding a database read as files is never read live. Whatever happens, each container is put back and checked. If a run is killed part way, the next run or backup resume puts back any stack it left paused or stopped.

After the snapshots restic keeps the newest of 7 days, 4 weeks and 6 months for each stack (keep_daily, keep_weekly and keep_monthly under [backup]), prunes the rest and checks the repository. Once a week the run also does the restore test. restic reads back a random 5% of the stored data against its checksums, and one stack's newest snapshot under 1 GiB is restored into a throwaway volume, a different stack each week. backup restore-test runs it now. A run or a restore test that finishes with something failed exits 1, so the timer's unit and your scripts see it.

backup restore <stack> puts a stack's volumes and folders back as a snapshot has them, the newest unless --snapshot names one. It's a severe change, so you type the stack's name. Each path is restored first into a staging folder seawall has just made, and what was there before is kept aside. volumes clean-aside removes those copies once you no longer need them. In the Backups tab S runs the data backup. On a stack's data row t runs the restore test and R restores it after showing its plan. On a dump row b runs db backup and t db restore-test.

With runtime.owner set, the owner's rootless stacks go to a repository of their own beside root's, <destination>.<owner>, with its own password. The restore test runs on each.

An S3 bucket

export S3_KEY=... S3_SECRET=...
seawall backup setup --to s3:https://s3.example.com/my-bucket/seawall --key-env S3_KEY --secret-env S3_SECRET --yes
seawall backup setup --to s3:https://nas.example.net:9000/backups --key-env S3_KEY --secret-env S3_SECRET --ca-file nas-ca.pem --yes

The repository can live in any S3-compatible bucket as well as a folder. Only https is taken, and the endpoint, bucket and path are checked. The endpoint can be a name or an address on your network, as a NAS or MinIO at home often is. Loopback, link-local and multicast addresses are refused. A bucket that isn't there yet is made.

The key and secret come from the environment variables you name. They go into backup-s3-credentials beside seawall.toml (0600) and never into an argument, the environment of a run or a log. restic gets the file mounted read-only, and the config capture's secret scan knows both values. For an endpoint with its own CA, --ca-file takes its certificate (PEM, never a key) and keeps a copy beside the credentials, 0600.

Setup gives the endpoint 90 seconds. If restic is still retrying then, setup stops and says why, such as a name that doesn't resolve, a key or secret the endpoint refuses or a certificate it can't trust.

restic reaches the bucket through a network of its own, seawall-backup. Its bridge is kept apart from every other bridge, setup adds a ufw rule that denies it this host and seawall-backup-egress.service drops anything it sends that doesn't leave by the default route. So it reaches the internet and your network but no port this host publishes. Setup is refused while ufw is off. Run, ls, the restore test and restore work against the bucket as they do against a folder.

Note

A rootless owner's stacks stay on a local destination for now, so they aren't backed up to a bucket.

An SFTP server

seawall backup setup --to sftp:[email protected]:/srv/restic
seawall backup setup --to sftp:[email protected]:/srv/restic --yes

The repository can live on an SFTP server too, as sftp:<user>@<host>[:<port>]:/<path>. The first run makes an ed25519 key (backup-sftp-key, 0600) and shows its public half. Add that line to the user's authorized_keys on the server and run setup again.

Setup reads the server's host key through restic's own network and shows its fingerprint. On a terminal you type the first 16 characters of the fingerprint as the server's own ssh-keygen -lf prints it. Off a terminal --host-key SHA256:... gives it whole. The key is pinned in backup-sftp-known-hosts (0600). restic's ssh reads no config, uses that key and host key alone, forwards nothing and never prompts. It goes out on the same seawall-backup network as S3.

Warning

If the server's host key changes, the run stops and says so, with the start of the pinned and the shown fingerprints. Check the new one with the server's admin, never by the one shown, then give it to backup setup ... --host-key. On a terminal you also type the host's name. The journal keeps the old and the new key.