Skip to content

M7-22 Audit log checkpoints in object storage

The brief names these rows D-M7-22-1 to D-M7-22-4; they follow the numbering of this log (M7-22/2 to M7-22/5), and M7-22/1 and M7-22/6 are rulings of the build.

No Date Ruling Reason
M7-22/0 2026-10-16 The ticket was built on an integration base and ported onto main with only its own commits; the English docs and spec of main win, and text this ticket adds is English (chapter and ADR names of main). The tickets are built in parallel on integration bases; main takes each ticket as one pull request; merged names win (D-W-100).
M7-22/1 2026-10-14 Base int/m1-wave17 at e2f9eb4, branch feat/M7-22-audit-checkpoints-s3 in its own worktree; no local merge, no push and no pull request from this build, the PR lane lands it. The brief names the English chapter files of main (19-security.md, 12-files.md, 03-data-model.md, 20-privacy.md); on this base they are 19-security.md, 12-files.md, 03-data-model.md and 20-privacy.md, and text this ticket adds there is English. Section 8 of docs/operations/backup-restore.md, which the brief builds on and which this base lacked, comes from main as its own commit. The process rulings D-W-100 and D-W-125 of main are not ported: they change no code. Instructions of the launcher, which override the brief's landing step; the brief's names follow main (D-W-100: the merged names win).
M7-22/2 2026-10-14 Test service (D-M7-22-1): the SeaweedFS image the files driver pins (chrislusf/seaweedfs:4.48 by digest, M3-02/13). Its S3 gateway supports bucket Object Lock with a default retention in mode COMPLIANCE (CreateBucket with ObjectLockEnabledForBucket, PutObjectLockConfiguration, GetObjectLockConfiguration; in a manual probe against the pinned image a locked version answered DeleteObject with its version ID with AccessDenied) and conditional writes (PutObject with If-None-Match: * on an existing key answers 412 PreconditionFailed), shown against the pinned image by TestLockCheck, TestConformance, TestObjectKeys (retention mode of a stored object) and TestOverwriteKeepsFirstVersion; the SeaweedFS wiki page "Amazon S3 API" (https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API, retrieved 2026-10-14) lists GetObjectLockConfiguration and PutObjectLockConfiguration as supported and links "S3 Object Lock and Retention" and "S3 Conditional Operations" (If-None-Match). No second test service is pinned. The tests read the image from internal/files/adapters/s3/seaweed_test.go, so Renovate moves one pin. The tests of adapters/httpapi create locked buckets with weed shell (s3.bucket.create -withLock) inside the container and need no S3 client. Design note 1: one free, pinned service with both features keeps one pin and the version the stack runs; the negative case "no lock" runs on the same service with a bucket without Object Lock. A real cloud bucket is out of scope (D-W-125, no cost).
M7-22/3 2026-10-14 Write once and key form (D-M7-22-2, D-M7-22-3): the object is [<prefix>/]<instance_id>/<seq with 20 digits, leading zeros>.jws, the layout of the file sink. Put reads the oldest version of the key first (the same bytes count as stored, other bytes are ErrCheckpointConflict), then writes with If-None-Match: * and an SHA-256 checksum; on 412 or 409 of a concurrent writer the stored bytes decide the same way. List pages through ListObjectVersions below [<prefix>/]<instance_id>/ (page size 1000, the tests 10), takes the oldest version of every checkpoint key, skips delete markers and other names, and returns the checkpoints in seq order. The brief suggested checkpoints/<seq>.jws; the instance ID keeps two instances in one bucket apart, as the file sink does, and the same width keeps the key order the seq order. A service that ignores the header only adds a version on a locked bucket, and the oldest version is the one signed first; a bucket with a default retention requires an integrity header on uploads.
M7-22/4 2026-10-14 New keys (D-M7-22-4): audit.checkpoint_s3_endpoint (empty, required with s3://), audit.checkpoint_s3_region (us-east-1), audit.checkpoint_s3_use_path_style (true), audit.checkpoint_s3_access_key_ref and audit.checkpoint_s3_secret_key_ref (file:<secrets.dir>/audit_checkpoint_s3_access_key and ..._secret_key), all chapter 19; no time or size key of its own: the egress client of purpose audit_checkpoint keeps egress.timeout and egress.max_response_bytes. retention.audit (365d, at least 1d, chapter 20) enters the schema with this first reader. The secret catalogue rows are supplied and go to the service server only. The cross-check audit_checkpoint_sink validates the endpoint, region, path style and the allow_private entry of an http endpoint at load (the rules of file.s3.*, copied into s3sink.Config across the module boundary). The SDK is allowed in internal/audit/adapters/s3sink (module audit gets adapter_vendors: [awssdk], depguard and TestS3SDKOnlyInAdapter name the package); its interface holds only PutObject, GetObject, ListObjectVersions and GetObjectLockConfiguration (TestAPIMethodSet). Own endpoint and credentials separate from file storage and backup (runbook 8 item 1). The brief also named migrate for the secret files "as the job runs there"; the job audit.chain_verify runs only in the server (roles worker and all), and bothy migrate never reads the keys, so the narrower list follows least privilege.
M7-22/5 2026-10-14 Lock check: the target passes only with Object Lock enabled and a default retention in mode COMPLIANCE whose period (days times 24 hours plus years times 365 days) is at least retention.audit; GOVERNANCE, a lock without default retention, a bucket without Object Lock and a shorter period fail with s3sink.ErrNotImmutable, naming audit.checkpoint_sink (and retention.audit for the period), never the endpoint. bothy config check runs it for an s3:// sink with or without --secrets (it needs the keys and one request; --secrets reports the two references as skipped unless the sink is s3://), and the start step audit_log builds the sink through the same wire.CheckpointSink and stops the start. Brief item 15 and runbook 8: Governance can be lifted with s3:BypassGovernanceRetention; a year counted as 365 days never overstates a lock.
M7-22/6 2026-10-14 The selection of the sink by scheme stays in internal/wire (wire.CheckpointSink, which Audit.BindLog now receives), not in internal/audit/app as the brief's scope lists it; the conformance suite is internal/audit/app/sinktest (only tests import it, TestFilestestNotInBinary also holds it out of the binary). app may not import adapters (go-arch-lint, ADR 0008); the M1-50 code already selected the sink in wire (D-W-100: follow the merged code).
M7-22/7 2026-10-15 s3sink.Sink keeps the body of every version it read in memory, by key and version ID (a version never changes; objects without a version ID are never cached), and reads the bodies a List has not seen with at most 16 concurrent GetObjects. A process reads each checkpoint once; every later run of audit.chain_verify costs one ListObjectVersions pass and one GetObject per new checkpoint (TestListScalesWithManyCheckpoints). Red team NEIN: the job lists the sink on every run, and one sequential GetObject per checkpoint against a remote store at 40 ms per request passed the one-minute job timeout of River after about 1400 checkpoints, about two months at audit.checkpoint_interval = 1h; from then on no checkpoint was signed. The cache holds a few hundred bytes per checkpoint.