| D-M7-02-0 |
2026-10-20 |
Base int/m1-wave46 at 9c05469 with feat/M3-03-upload-sessions at 0b60fd7 merged first (stacked build), branch feat/M7-02-backup-service in its own worktree; not merged locally, no pull request; the full verification runs on the build branch build/M7-02. The base names the chapters in German (16-betrieb.md, 03-datenmodell.md, 02-architektur.md) and has no section 10 in this file, so the rows stand in a section of the ticket and added text is English. The brief relies on no core rule or D-W ruling the base lacks (R-OP-06 to R-OP-13 and 03 4.12 are on the base; D-W-100 and D-W-125 are rulings about writing briefs), so nothing is ported first. Names that differ from the brief (D-W-100): the migration is 00049_status_backup_runs.sql, the image lives in services/server/images/bothy-postgres/. |
Instructions of the launcher; merged names win (D-W-100). |
| D-M7-02-1 |
2026-10-20 |
Cron: an own five-field parser in internal/status/domain (ParseSchedule, Matches, Next), no library: *, numbers, ranges, lists, */n and a-b/n; day of week 0 to 7 (0 and 7 Sunday); when day of month and day of week are both restricted, either matches. Schedules are evaluated in UTC. |
Five fields with this grammar cover every backup.* default; a library would add a vendor and the license gate for a hundred lines; UTC because every instant of the server is UTC (03 principle 2). |
| D-M7-02-2 |
2026-10-20 |
Scheduler: once a minute the kinds due in (previous tick, now] start; the first window starts at the start of the process, so missed times while it was down are not caught up and the next due time runs. pg_full and pg_diff share one slot ("physical"); a full and a differential due in the same window run only the full; a kind whose slot runs is skipped, not queued. bothy-backupd backup <kind> runs one kind now and exits 0 or 1. |
Brief design note 1; pgBackRest holds one backup lock per stanza, so two physical runs at once would fail anyway. |
| D-M7-02-3 |
2026-10-20 |
Role bothy_backup (03 4.12, design note 3): created by migrate up --bootstrap like the four roles of 16 7.1 with LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS, CONNECT and USAGE on public; the migration grants it SELECT, INSERT, UPDATE on backup_runs and nothing else; bothy_app gets SELECT (console and status page read the table, 03). Secret db_backup_password (generated, to migrate and backup), key migrate.backup_password_ref; the Compose service backup connects with it as database.user. The manifest queries run as migrate.bootstrap_user in one READ ONLY transaction with REPEATABLE READ and an exact count(*) per ordinary or partitioned table of public (partitions through their parent), in a pool without statement deadline. |
Least privilege for the writer of the row; one snapshot for every count; exact counts because the restore test compares them. |
| D-M7-02-4 |
2026-10-20 |
Off-site targets: only S3-compatible buckets with versioning and Object Lock, backup.offsite.target = s3:https://<host>[:<port>]/<bucket>[/<prefix>], new keys backup.offsite.region (us-east-1) and backup.offsite.use_path_style (true); in the bucket repo2 at <prefix>/pgbackrest, the restic repository at <prefix>/restic, the object mirror at <prefix>/objects/. The key of backup is offsite/access_key_id and offsite/secret_access_key (plus offsite/ca.crt for a private CA); bothy-backupd passes repo2 to its pgBackRest children as PGBACKREST_REPO2_* variables with PGBACKREST_EXPIRE_AUTO=n, and expire runs only on repo1. The owner's expiry is bothy-backupd expire --offsite --credentials DIR with a separate key that never enters the volume secrets (runbook 9). restic's REST server and SFTP are not supported. |
pgBackRest's repo2 cannot use restic's REST server, and restic over SFTP needs an SSH client the image does not have (the brief allows only restic as a new binary); one target kind for all three tools keeps one key and one proof. Append-only follows from Object Lock, not from the tools: the probe and TestBackupKinds show that SeaweedFS refuses DeleteObject with a version id to the key of backup. |
| D-M7-02-5 |
2026-10-20 |
Object backup of the driver s3: always a mirror (adapters/s3mirror) of the file bucket into <prefix>/objects/ of the off-site bucket: list, HeadObject, then GetObject and PutObject with If-None-Match: * and a SHA-256 checksum for every missing key; it never deletes or overwrites, a deleted source object stays. A provider replication mode that only checks the replica is not built; the driver s3 without an off-site target fails its object run with s3_mirror_not_configured. |
Brief design note 5; checking a provider's replication needs a provider API this build cannot test for free (D-W-125), and a mirror works with every S3 service. |
| D-M7-02-6 |
2026-10-20 |
Manifest: built after the backup step from the snapshot of D-M7-02-3 and the references (pgbackrest.label of repo1 for pg_full and pg_diff, restic.snapshot_id for logical and objects with the driver fs); stored in backup_runs.manifest and as a restic snapshot of stdin (manifest-<run id>.json, tags manifest, <kind>, <run id>) in the restic repository (off-site, else local), which restic encrypts. A failed run stores no manifest and error is a code (<tool>_exit_<n>, manifest_store_failed and others). size_bytes is what the run added: repo1's repository delta, restic's data_added_packed, or the bytes the mirror copied. Without an off-site target the local restic repository (/var/lib/pgbackrest/restic) is pruned after each run with backup.logical.retention and backup.objects.retention (manifests with the larger of both). |
A row of the same table and an encrypted copy next to the backup, as backup-restore.md 3 asks; restic is the one tool here that encrypts arbitrary files. The counts describe the database at the end of the run, after the backup; M7-03 compares them knowing that writes between the end of the backup and the snapshot may differ. |
| D-M7-02-7 |
2026-10-20 |
bothy.backup.wal.last_archived.timestamp comes from pg_stat_archiver.last_archived_time (superuser connection), not from pgbackrest info. The three bothy.backup.restore_test.* instruments stay to M7-03; bothy.backup.duration and bothy.backup.size already accept restore_test in their closed attribute set. Metrics go through otel.SetupMetrics (new) with service.name = bothy-backupd and without a pseudonym key. |
pgbackrest info reports WAL ranges but no archive time; pg_stat_archiver is the server's own record of the last archived segment. bothy-backupd holds no pseudonym key and records no person. |
| D-M7-02-8 |
2026-10-20 |
bothy migrate up (16 10 item 1, R-OP-13): before River and goose run, with migrate.require_recent_backup, an existing schema (version above 0) with pending goose migrations needs a succeeded pg_full, pg_diff or logical run whose started_at is at most migrate.recent_backup_max_age old; otherwise exit 1 with the hint bothy-backupd backup pg_full. Exempt: an empty database, no pending migration, and a schema without the table backup_runs (it predates M7-02, so no run can exist; one line on stdout says so). |
Brief rule 17; without the exemption the first upgrade to this version could never pass. |
| D-M7-02-9 |
2026-10-20 |
bothy-backupd loads the full server schema (wire.ConfigSchema), so the image builds with api/ and migrations/ in its context; it leaves the image variables BOTHY_PGBACKREST_REPO_HOST, BOTHY_PGBACKREST_PG_HOST, BOTHY_PGBACKREST_REPO2 and BOTHY_PG_ARCHIVE_TIMEOUT out of the configuration sources. New image variable BOTHY_PGBACKREST_REPO2=on (Compose: set when BOTHY_BACKUP_OFFSITE_TARGET is set) makes the database host name repo2 on the repository host, so archive-push writes every segment to both repositories. The image creates /var/lib/bothy-backup for the user 999. |
One schema means bothy config check and the service read the same keys; the image variables share the prefix BOTHY_ but are read by the entry point, and R-OP-04 would end the start for them. |
| D-M7-02-10 |
2026-10-20 |
The restic binary of bothy-postgres carries the file capability cap_dac_read_search=+ep (set in the build stage, kept by the copy), and the Compose service backup gets cap_add: [DAC_READ_SEARCH]: the volume files belongs to the server's user 65532 with mode 0700 (files 0600), which the file system driver requires, and backup runs as 999. Only restic gets the capability; bothy-backupd, pgBackRest and pg_dump run without it. Without cap_add the kernel refuses to start restic, so a stack that forgets it fails its logical and object runs loudly. |
restic's documented way to back up files of another user without running as root; the server keeps its ownership rule, and the entry point keeps refusing root. |
| D-M7-02-11 |
2026-10-20 |
The executor orders the restic calls of one process: backups (and the manifest store) share a lock, forget --prune and the first init hold it alone. |
restic refuses a prune while another restic holds a lock; without the order a local prune after a logical run fails a concurrent object backup or manifest store, or the other way round. A second process (a bothy-backupd backup by hand during a scheduled run) is not ordered; its failure is recorded as a code and retried by the next run. |
| D-M7-02-12 |
2026-10-20 |
The owner's off-site expiry (bothy-backupd expire --offsite) does not delete object versions itself. The runbook requires a lifecycle rule on the bucket prefix that expires noncurrent versions after NoncurrentDays of at least the Object Lock period and removes expired delete markers; restic forget --prune and pgbackrest expire on repo2 only add delete markers, and the rule removes the old versions later. Runbook 2 and 9 and 16 8.2 say so; off-site data lives up to its retention plus NoncurrentDays. Found by the red team: without the rule nothing ever left the versioned bucket while the command exited 0. |
A command that deleted every noncurrent version with the owner's key would also make permanent whatever delete markers a taken-over backup key had set (its key may add delete markers for restic's lock files); the delay of the rule leaves the owner time to notice. A bucket rule is the provider's own mechanism and works without the owner running anything. |
| D-M7-02-13 |
2026-10-22 |
Port onto main from the ticket branch (own commits ed8648a..b0f4491, after the merge of M3-03, which is on main as #223) as one change: the English text of main wins where both sides changed a document, and the ticket's additions to the German chapters are placed in their English counterparts (16-operations.md, 12-files.md, ADR 0010 and 0048). The migration of backup_runs is 00050 on main, after 00048 of M2-03 (#230) and 00049 of M6-01 (#197); the image keeps the configuration link of D-M7-01-9 and adds only the scratch directory and restic; the sqlc models of status are generated again. The test helper that copies the embedded migrations lives in migrate_backup_test.go, because main has no such helper. |
main has no integration branch; the rules for a port bring only the ticket's own commits and adapt them to main. |
| D-M7-02-14 |
2026-10-22 |
Found by the pull request review: pg_dump connects with database.sslmode and database.sslrootcert as PGSSLMODE and PGSSLROOTCERT; verify-full without a CA file uses PGSSLROOTCERT=system, the system roots the pool of the server uses (libpq 16 and later). The rulings of this ticket moved to docs/decisions.d/M7-02.md with the fragment change (PL/24); references in code and documents name this file. |
Without the variables libpq's default prefer would send the dump of an external database without certificate check, or in clear text (SEC-TLS-5). |