| D-M7-03-0 |
2026-10-20 |
Base int/m1-wave49 at eb15f40, branch feat/M7-03-restore-test in its own worktree; not merged locally, no pull request; the full verification runs on the build branch build/M7-03. The base names the chapters in German (16-betrieb.md, 03-datenmodell.md, 02-architektur.md, 19-sicherheit.md) and has no section 10 in this file, so the rows stand in a section of the ticket and added text is English. The brief relies on no core rule or D-W ruling the base lacks (R-OP-10 and backup_runs.checks of 03 4.12 are on the base; D-W-100 and D-W-125 are rulings about writing briefs), so nothing is ported first. Names that differ from the brief (D-W-100): the dependency table of 02 is section 4.4 "Allowed dependencies" (4.5 holds the kernel ports), the subcommand is bothy-backupd restore-test, the scratch directory is /var/lib/bothy-backup/restore, the image test lives in services/server/test/postgresimage/restore_test.go (TestRestoreTest). |
Instructions of the launcher; merged names win (D-W-100). |
| D-M7-03-1 |
2026-10-20 |
Order and skipping (design note 1): check pgbackrest reads the newest succeeded pg_full or pg_diff from backup_runs, restores its label, runs pgbackrest verify on repo1 and starts the copy. No backup (no_backup), a manifest that does not parse (manifest_invalid), a failed restore (the exit code of pgBackRest) or a failed start (scratch_start_failed) fail check 1 and skip checks 2 to 6 with restore_failed; a failed verify fails check 1 with its code, but the copy is still checked. A check that panics fails with check_panicked and the others run. duration runs always: from the start of the row to the end of the clearing. A skipped check fails nothing; the row is failed with checks_failed when a check failed and with scratch_cleanup_failed when the directory could not be emptied. The test has its own slot restore_test and runs next to the backups. |
Only a missing copy makes checks 2 to 6 meaningless; a damaged repository and a sound copy are two findings. Its own slot keeps a long test from skipping a scheduled backup. |
| D-M7-03-2 |
2026-10-20 |
Audit port (design note 2): the root package audit gets RowsVerifier with RowsInput (instance id, public keys of server_signing_keys, iterators over audit_events and audit_checkpoints in seq order); app.RowsVerifier and app.VerifyRows implement it with the walk of audit.verify (domain.Walk from row 1 with ZeroHash, no anchors yet) and the same verdict function as Log.Verify (missing checkpoint row, first invalid checkpoint, row below 1, smallest seq wins). The checkpoints come from audit_checkpoints of the copy and are trusted like those of the sink: JWS typ audit-checkpoint+json, signature of the key its kid names among the supplied keys of purpose audit_checkpoint with the same algorithm, payload of the instance naming the stored seq and hash, links (domain.CheckCheckpoints). status reads the rows of the copy over its own pool (adapters/postgres.ScratchDB) and never runs SQL on audit tables of production; modules.yaml, 02 4.4 and the component diagram name the edge status to audit. |
Owner rule 2: one implementation of the chain verification. The copy has no sink of its own; its audit_checkpoints are the signed record it carries. |
| D-M7-03-3 |
2026-10-20 |
Manifest check: the manifest of a physical run is read after the backup step (D-M7-02-6), the copy ends at the consistency point of the backup (recovery target immediate), so writes in between may differ. The check fails when schema version, instance_id or sync_epoch differ, when the copy is newer than its manifest (event head or audit head above it, copy_newer_than_manifest), when the audit heads are equal but the hashes differ, or when a table of the manifest is missing or the copy has another one. When the event head and the audit head of the copy equal those of the manifest, the row counts of event_log and audit_events must be equal (row_counts_differ); otherwise no count is compared. The counts of the other tables are never compared: sessions, tokens, cleanup jobs and backup_runs itself change rows without an event or audit row while a backup runs (found by the code review). |
Exact counts on a busy instance would fail tests of undamaged backups and page the owner; every row of the two logs carries the number of its log, so at equal heads an undamaged copy holds exactly the rows of its manifest. |
| D-M7-03-4 |
2026-10-20 |
Sampling (rule 15): per test ORDER BY random() LIMIT n over threads and over files with deleted_at IS NULL AND upload_state = 'complete' AND scan_status IN ('clean', 'skipped'); with fewer rows every row is checked. A pending upload has no object at its key yet, and the scan pass deletes the object of an infected or unscannable file while its row stays (found by the red team), so both are left out. Thread rule: count(*) = max(thread_seq) - min(thread_seq) + 1, min >= 1 and max = threads.last_message_seq, or no messages and last_message_seq = 0. Objects: restic restore latest --tag objects --path <file.fs.root> with one --include per key into <scratch>/objects (the path picks the object backup, because the manifest snapshot of an objects run carries the tag objects too), read through an os.Root; a key the backup lacks is object_missing, other bytes object_hash_mismatch. With the driver s3 the objects lie in the mirror of the off-site bucket, which this build does not fetch: check 6 is skipped with objects_driver_s3. |
The brief names the restic fetch; a fetch from the mirror needs an S3 client in the restore path, left for the ticket that needs it (follow-up in the report). |
| D-M7-03-5 |
2026-10-20 |
Test hook (design note 3): only the build tag restoretesthook compiles adapters/scratch/hook_restoretesthook.go; release builds compile the empty hooks of hook.go. The image test builds its own bothy-backupd with the tag and runs restore-test through it inside the container backup; RESTORE_TEST_HOOK (no BOTHY_ prefix, so R-OP-04 never sees it) selects event_log_gap, audit_hash or object_bytes; every run of the tagged build writes /proc/net/tcp and /proc/net/tcp6 while the copy runs to restore-test-tcp next to the scratch directory (/var/lib/bothy-backup, mode 0700). |
A test-only build keeps the manipulation out of every shipped binary and still tests the real image. |
| D-M7-03-6 |
2026-10-20 |
A deleted event_log row fails check event_log (event_log_gap) and check manifest (row_counts_differ), because the copy is quiet and its event_log count is one below the manifest; the image test expects both, the other five pass. The audit hash and the object bytes fail exactly their check. |
A gap inside a partition always changes the count or the bounds of event_log, so "exactly its check" (design note 3) cannot hold for this case while check 2 compares row counts as 16 8.3 item 2 asks. |
| D-M7-03-7 |
2026-10-20 |
Scratch PostgreSQL: restored with --repo=1 --reset-pg1-host --pg1-path=<scratch>/pgdata --type=immediate --target-action=promote --archive-mode=off, restore_command overridden with the same options; started by its absolute path with listen_addresses empty, port=5432 (the socket file name), the socket in <scratch>/socket with mode 0700, its own pg_hba.conf (local all all trust), archive_mode=off, ssl=off, max_parallel_workers_per_gather=0; other settings, such as max_connections, stay those of the copy (recovery refuses lower values). Stopped with SIGINT, with SIGKILL when the fast shutdown does not end in time, then the directory is emptied. |
The repository host has no database role of its own in pgBackRest; promotion makes the copy writable for the test hook only, and archiving off keeps its new timeline out of the repository. The socket mode limits access to the user of bothy-backupd. Parallel query would need the shared memory of a database container. |
| D-M7-03-8 |
2026-10-20 |
Metrics: bothy.backup.restore_test.duration and bothy.backup.duration{bothy.backup.kind=restore_test} after every test, no bothy.backup.size; bothy.backup.restore_test.last_success.timestamp after a test without a failed check and without a cleanup failure, primed from backup_runs at start; bothy.backup.restore_test.failures plus one per failed check with bothy.reason the check name (closed set of the seven names). |
alerting.md: one failure ticket, two in a row or an audit chain error page, BothyRestoreTestStale on the last success. |
| D-M7-03-9 |
2026-10-20 |
Configuration: backup.restore_test.schedule (string, also in the cross-check backup_schedules), backup.restore_test.sample_files and backup.restore_test.sample_threads (int, at least 1) and dr.rto_target (duration, at least 1s) become active keys; backup.restore_test.max_age and dr.rpo_target stay planned, because only the alert rules read them. |
R-OP-04: a key enters the schema with its first consumer. |
| D-M7-03-10 |
2026-10-22 |
Port onto main from the ticket branch (own commits eb15f40..a582b38, after M7-02, which is on main as #239) as one change: the English text of main wins where both sides changed a document, and the ticket's additions to the German chapters are placed in their English counterparts (16-operations.md 8.3, 02-architecture.md 4.4, c4-3-server-components.mmd); the configuration keys name the chapter 16-operations.md; the ruling rows and the changelog entry are fragments (PL/24). The image test of M7-04 that shared the file name restore_test.go on the integration branch is not on main, so nothing is renamed. |
main has no integration branch; the rules for a port bring only the ticket's own commits and adapt them to main. |
| D-M7-03-11 |
2026-10-22 |
A restore test whose context ends while it runs (bothy-backupd stops) finishes its row as failed with the code canceled, like a canceled backup run (M7-02), still clears the scratch area and records no bothy.backup.restore_test.failures. Two restore tests of separate processes at once (the daemon and bothy-backupd restore-test by hand) are not excluded across processes: the slot is process-local, as for the backup kinds, and the runbook runs the command by hand outside the schedule. |
Found by the pull request review: a container restart during a test would otherwise fail its remaining checks and raise a failure alert without a damaged backup. A cross-process lock would guard an operator error only, with no reproduced harm. |