M7-22 Audit log checkpoints in object storage¶
The brief names these rows D-M7-22-1 to D-M7-22-4; they follow the numbering of this log (M7-22/2 to M7-22/5), and M7-22/1 and M7-22/6 are rulings of the build.
| No | Date | Ruling | Reason |
|---|---|---|---|
| M7-22/0 | 2026-10-16 | The ticket was built on an integration base and ported onto main with only its own commits; the English docs and spec of main win, and text this ticket adds is English (chapter and ADR names of main). | The tickets are built in parallel on integration bases; main takes each ticket as one pull request; merged names win (D-W-100). |
| M7-22/1 | 2026-10-14 | Base int/m1-wave17 at e2f9eb4, branch feat/M7-22-audit-checkpoints-s3 in its own worktree; no local merge, no push and no pull request from this build, the PR lane lands it. The brief names the English chapter files of main (19-security.md, 12-files.md, 03-data-model.md, 20-privacy.md); on this base they are 19-security.md, 12-files.md, 03-data-model.md and 20-privacy.md, and text this ticket adds there is English. Section 8 of docs/operations/backup-restore.md, which the brief builds on and which this base lacked, comes from main as its own commit. The process rulings D-W-100 and D-W-125 of main are not ported: they change no code. |
Instructions of the launcher, which override the brief's landing step; the brief's names follow main (D-W-100: the merged names win). |
| M7-22/2 | 2026-10-14 | Test service (D-M7-22-1): the SeaweedFS image the files driver pins (chrislusf/seaweedfs:4.48 by digest, M3-02/13). Its S3 gateway supports bucket Object Lock with a default retention in mode COMPLIANCE (CreateBucket with ObjectLockEnabledForBucket, PutObjectLockConfiguration, GetObjectLockConfiguration; in a manual probe against the pinned image a locked version answered DeleteObject with its version ID with AccessDenied) and conditional writes (PutObject with If-None-Match: * on an existing key answers 412 PreconditionFailed), shown against the pinned image by TestLockCheck, TestConformance, TestObjectKeys (retention mode of a stored object) and TestOverwriteKeepsFirstVersion; the SeaweedFS wiki page "Amazon S3 API" (https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API, retrieved 2026-10-14) lists GetObjectLockConfiguration and PutObjectLockConfiguration as supported and links "S3 Object Lock and Retention" and "S3 Conditional Operations" (If-None-Match). No second test service is pinned. The tests read the image from internal/files/adapters/s3/seaweed_test.go, so Renovate moves one pin. The tests of adapters/httpapi create locked buckets with weed shell (s3.bucket.create -withLock) inside the container and need no S3 client. |
Design note 1: one free, pinned service with both features keeps one pin and the version the stack runs; the negative case "no lock" runs on the same service with a bucket without Object Lock. A real cloud bucket is out of scope (D-W-125, no cost). |
| M7-22/3 | 2026-10-14 | Write once and key form (D-M7-22-2, D-M7-22-3): the object is [<prefix>/]<instance_id>/<seq with 20 digits, leading zeros>.jws, the layout of the file sink. Put reads the oldest version of the key first (the same bytes count as stored, other bytes are ErrCheckpointConflict), then writes with If-None-Match: * and an SHA-256 checksum; on 412 or 409 of a concurrent writer the stored bytes decide the same way. List pages through ListObjectVersions below [<prefix>/]<instance_id>/ (page size 1000, the tests 10), takes the oldest version of every checkpoint key, skips delete markers and other names, and returns the checkpoints in seq order. |
The brief suggested checkpoints/<seq>.jws; the instance ID keeps two instances in one bucket apart, as the file sink does, and the same width keeps the key order the seq order. A service that ignores the header only adds a version on a locked bucket, and the oldest version is the one signed first; a bucket with a default retention requires an integrity header on uploads. |
| M7-22/4 | 2026-10-14 | New keys (D-M7-22-4): audit.checkpoint_s3_endpoint (empty, required with s3://), audit.checkpoint_s3_region (us-east-1), audit.checkpoint_s3_use_path_style (true), audit.checkpoint_s3_access_key_ref and audit.checkpoint_s3_secret_key_ref (file:<secrets.dir>/audit_checkpoint_s3_access_key and ..._secret_key), all chapter 19; no time or size key of its own: the egress client of purpose audit_checkpoint keeps egress.timeout and egress.max_response_bytes. retention.audit (365d, at least 1d, chapter 20) enters the schema with this first reader. The secret catalogue rows are supplied and go to the service server only. The cross-check audit_checkpoint_sink validates the endpoint, region, path style and the allow_private entry of an http endpoint at load (the rules of file.s3.*, copied into s3sink.Config across the module boundary). The SDK is allowed in internal/audit/adapters/s3sink (module audit gets adapter_vendors: [awssdk], depguard and TestS3SDKOnlyInAdapter name the package); its interface holds only PutObject, GetObject, ListObjectVersions and GetObjectLockConfiguration (TestAPIMethodSet). |
Own endpoint and credentials separate from file storage and backup (runbook 8 item 1). The brief also named migrate for the secret files "as the job runs there"; the job audit.chain_verify runs only in the server (roles worker and all), and bothy migrate never reads the keys, so the narrower list follows least privilege. |
| M7-22/5 | 2026-10-14 | Lock check: the target passes only with Object Lock enabled and a default retention in mode COMPLIANCE whose period (days times 24 hours plus years times 365 days) is at least retention.audit; GOVERNANCE, a lock without default retention, a bucket without Object Lock and a shorter period fail with s3sink.ErrNotImmutable, naming audit.checkpoint_sink (and retention.audit for the period), never the endpoint. bothy config check runs it for an s3:// sink with or without --secrets (it needs the keys and one request; --secrets reports the two references as skipped unless the sink is s3://), and the start step audit_log builds the sink through the same wire.CheckpointSink and stops the start. |
Brief item 15 and runbook 8: Governance can be lifted with s3:BypassGovernanceRetention; a year counted as 365 days never overstates a lock. |
| M7-22/6 | 2026-10-14 | The selection of the sink by scheme stays in internal/wire (wire.CheckpointSink, which Audit.BindLog now receives), not in internal/audit/app as the brief's scope lists it; the conformance suite is internal/audit/app/sinktest (only tests import it, TestFilestestNotInBinary also holds it out of the binary). |
app may not import adapters (go-arch-lint, ADR 0008); the M1-50 code already selected the sink in wire (D-W-100: follow the merged code). |
| M7-22/7 | 2026-10-15 | s3sink.Sink keeps the body of every version it read in memory, by key and version ID (a version never changes; objects without a version ID are never cached), and reads the bodies a List has not seen with at most 16 concurrent GetObjects. A process reads each checkpoint once; every later run of audit.chain_verify costs one ListObjectVersions pass and one GetObject per new checkpoint (TestListScalesWithManyCheckpoints). |
Red team NEIN: the job lists the sink on every run, and one sequential GetObject per checkpoint against a remote store at 40 ms per request passed the one-minute job timeout of River after about 1400 checkpoints, about two months at audit.checkpoint_interval = 1h; from then on no checkpoint was signed. The cache holds a few hundred bytes per checkpoint. |