Skip to content

M1-15: Sequence counters and instance record

Status: ready | Depends on: M1-11, M1-13 | Milestone: M1

Goal

After this ticket the module instance owns the tables instance and sequence_counters, offers the port Counters that only the phase hooks of authz, sync and audit may use, records the instance identity, schema version and minimum server version during bothy migrate up, exposes the setup state, and keeps a server whose version is below instance.min_server_version from becoming ready. The counter rows are the serialization points that make event sequence, audit chain and permission version gapless and commit-ordered (B8, ADR 0009, 03 R-DM-05).

Read first

  • docs/spec/03-data-model.md, "Principles" (items 1, 2, 11, 12), "Instance (instance)" tables instance and sequence_counters, "Transaction rules" ("Global event sequence", "Lock order", "Sync epoch and restore"), "Decisions of this chapter" (R-DM-01, R-DM-05, R-DM-26).
  • docs/spec/04-sync-protocol.md, "Global, commit-ordered sequence" ("Pattern with a counter row", "Other counters").
  • docs/spec/02-architecture.md, "Module list" (instance), "Ports, events and sources per module" (Counters), "Allowed dependencies" (instance uses nothing), "Flow", "Lock order" (only the phase hooks lock counters, checked against the registrations), "Other checks in the gate", "Start" (steps 4 and 5), "Health", "Commands of the binary" (migrate sets instance.min_server_version).
  • docs/spec/16-operations.md, "Two listeners", "Database" ("Roles"), "Upgrade" (item 4).
  • docs/operations/upgrade.md, "Rollback" and "Schema check at startup" (check name schema_requires_newer_server).
  • docs/spec/08-identity.md, "Initial installation" (meaning of setup_completed_at).
  • docs/spec/18-release.md, "Versions" rules (file VERSION, SemVer without v).
  • ADR 0007, ADR 0009.
  • What earlier tickets built: internal/kernel/txn, internal/platform/pg unit of work and internal/wire registry plus golden file (M1-13); migrations/, cmd/bothy migrate, pgtest (M1-12); internal/platform/health readiness checks (M1-11); internal/instance/README.md.
  • docs/PLAN.md section "Way of working per ticket"; docs/quality/gates.md; CONTRIBUTING.md; docs/quality/red-team-gate.md.

Tables: instance, sequence_counters. No API operation and no action id.

Fixed requirements that apply

  1. No MVP. New from scratch and directly in the final state; stages are only the order.
  2. Strictly modular: clear module and service boundaries, dependencies only through defined interfaces, no cross access.
  3. Documentation in code: every public module, class, function and interface with a docstring/doc comment (purpose, parameters, return value, errors, example where useful); comments explain the why, not the what. API documentation is generated from code (OpenAPI from the routes, dartdoc, Python API docs in MkDocs).
  4. Documentation in the repo: README per module/service, ADR for every architecture decision, CHANGELOG per release, keep the documentation site current. A change without matching documentation is not done.
  5. Quality gates (SonarQube, linters, type checking, security scanners, test coverage) block the merge.
  6. Test first: write each test, run it and see it fail (keep the red output for the report), then write the code until it passes, then clean up.
  7. No en dash or em dash characters in anything you write (code, comments, docs, commit messages). Check every changed file with uv run --project tools --locked --quiet python tools/hooks/text_rules.py <file>; it must exit 0 (plain grep -P fails on macOS).
  8. White label: no company names, company or mail domains, person names, host names, real IP addresses or paths under a user home directory. Examples use example.com and the names Alex, Sam, Robin.
  9. Configuration values only through named keys that appear in docs/operations/configuration.md and the config schema in code (Ruling L4).
  10. Tables exactly as 03 defines instance and sequence_counters (columns, types, nullability, defaults, checks, keys, retention class config); the three counter rows event_seq, audit_seq, permission_version are created by the migration that creates the table (03).
  11. IDs are UUIDv7 generated by the application, never by the database (R-DM-01).
  12. Counter values only increase (03 "Rules / notes" of value); locking a counter is the last step before commit for event_seq and audit_seq and the first step for permission_version (03 "Lock order", 04 "Pattern with a counter row").
  13. Only the phase hooks of authz, sync and audit use Counters; no use case calls it (02 "Lock order").
  14. Only the module instance reads and writes its tables (03 principle 12).
  15. bothy migrate sets instance.min_server_version; a server with a smaller version does not become ready and /readyz names schema_requires_newer_server (02 "Commands of the binary", 16 "Upgrade" item 4, upgrade.md "Schema check at startup" item 2). A newer, compatible schema without this lock is allowed (rollback path, upgrade.md "Rollback").
  16. Without setup_completed_at only the setup path starts (02 "Start" step 5; the setup path itself is M1-37).

Scope

All Go paths are relative to services/server/.

  1. Migration (goose, SQL, in migrations/): tables instance and sequence_counters per 03; INSERT of the three counter rows with value 0; a BEFORE UPDATE trigger function instance_counter_monotonic() that raises when NEW.value < OLD.value or the name changes; grants for bothy_app: SELECT, UPDATE (value) on sequence_counters, SELECT on instance; no INSERT or DELETE for bothy_app on either table. Follow the grant pattern M1-12 set up. Name the file with the next free sequential number of the base branch (ADR 0007 rule 4); renumber before the final make check if another ticket took it.
  2. internal/instance (port):
    • type CounterName string with CounterEventSeq = "event_seq", CounterAuditSeq = "audit_seq", CounterPermissionVersion = "permission_version".
    • type Counters interface { Advance(ctx context.Context, name CounterName, n int64) (last int64, err error); Current(ctx context.Context, name CounterName) (int64, error) }. Advance locks the row until commit (UPDATE ... SET value = value + $n ... RETURNING value) and returns the new value, so the numbers are last - n + 1 .. last. Current reads the value without a lock (inside the hook that already holds the lock, or the committed value).
    • type Record struct { InstanceID ids.ID; SetupCompletedAt time.Time; SchemaVersion int64; MinServerVersion string; SyncEpoch ids.ID; CreatedAt time.Time } with func (r Record) SetupCompleted() bool (zero time means not completed; no JSON null anywhere later, ADR 0053).
    • type State interface { Load(ctx context.Context) (Record, error); RecordMigration(ctx context.Context, schemaVersion int64, minServerVersion string) (Record, error) }. Load returns not_found kind when the row is missing. RecordMigration creates the row if missing (new UUIDv7 for instance_id and sync_epoch) and sets schema_version and min_server_version; it never changes instance_id, sync_epoch or setup_completed_at.
  3. internal/instance/domain: counter name validation; phase rule (see Design notes); CompareSemVer(a, b string) (int, error) after SemVer 2.0.0 precedence including pre-release identifiers (https://semver.org/spec/v2.0.0.html, section 11), hand-written because the domain layer imports no third-party library; CheckServerVersion(own, min string) error.
  4. internal/instance/app: services implementing Counters and State over an outgoing repository port.
  5. internal/instance/adapters/postgres: sqlc queries under queries/, repository; entry in sqlc.yaml; generated code checked in.
  6. Server version: internal/platform/buildinfo.Version() from M1-10 (18 "Components" rule 5).
  7. Schema compatibility constant (18 "Components" rule 6, upgrade.md "Schema check at startup"): in the package that embeds migrations/ (M1-12), add const MinServerVersion = "<content of services/server/VERSION>" with a doc comment: the oldest server version allowed to run against the schema these migrations produce; per upgrade.md rule M1 it is the previous release, a deliberate break sets the own version; before the first release it equals the current version.
  8. cmd/bothy migrate up: after the migrations, if the database version equals the newest migration this binary knows, call State.RecordMigration(goose version, MinServerVersion); if the database is newer than this binary, leave the row unchanged and exit 0 (upgrade.md "Rollback" item 1). The write runs in a unit of work on the migrate connection.
  9. Start (internal/wire, internal/platform/lifecycle): after the schema check of M1-12, load the record (missing row: abort start with a message telling the instance operator to run bothy migrate up); keep it for later tickets (setup state, instance_id, sync_epoch); run CheckServerVersion(buildinfo version, record.MinServerVersion); on failure register a failing readiness check named schema_requires_newer_server with the API of platform/health, keep running, never become ready. Applies to all roles.
  10. Phase guard and injection: Counters is constructed only in internal/wire and passed only to phase hook constructors (none exist yet).
  11. Static check TestCountersOnlyUsedByPhaseHooks in internal/wire (see Tests).
  12. Docs: internal/instance/README.md (status, tables, Counters rule, State, start check); CHANGELOG. 02 "Lock order" and "Other checks in the gate" (phase guard, static check), 03 instance (row created by migrate up), 18 (version constants) and upgrade.md already say what this ticket builds.

Out of scope

  • instance_settings, feature_flags, server_signing_keys and their ports Settings, Flags, Keys (M1-46, M1-47, M1-32).
  • Setting setup_completed_at and the 503 setup_required behavior (M1-37); the setup path is only read here.
  • A new sync_epoch after restore (bothy restore finalize, M7).
  • The phase hooks that use Counters (M1-16 sync, M1-24 audit, M1-29 authz).
  • The start abort on an older schema (goose version check, M1-12).

Boundaries

  • A new migration needs its line in migrations.sha256: run make migrations-lock after creating the file and before make check; never edit a merged migration (M1-01, ADR 0007).
  • Work in a dedicated git worktree on the branch feat/M1-15-instance-counters.
  • Conventional Commits with scope (for example feat(instance): ...) and DCO sign-off (git commit -s).
  • Never push, tag, open pull requests or create remote repositories.
  • Do not modify files outside the scope above, except the docs and changelog entries named here.
  • Follow the host rules supplied by the launcher.
  • Stop only processes you started, by PID. Temporary directories only via mktemp -d, removed at the end. Test containers and listeners bind 127.0.0.1 and port 0 or a random port (P-09, P-10).
  • Never read or print secrets. No network calls except package downloads for pinned tools and Go modules.

Design notes

  • Phase guard (02 "Lock order": enforce the lock order at run time instead of relying on review): the Counters service checks txn.CurrentPhase(ctx): Advance(permission_version) only in rights_begin; Advance(event_seq) only in sync; Advance(audit_seq) only in audit; Current(...) only in one of the four hook phases. Any other call returns kind internal with a message naming counter and phase. n < 1 is an internal error. Unknown names are rejected in the domain before SQL; the table CHECK is the second line.
  • Static check: TestCountersOnlyUsedByPhaseHooks parses the Go packages under internal/ (golang.org/x/tools/go/packages if already in go.mod, else go/parser plus go/types) and lists every package that references instance.Counters or its methods. Allowed: internal/instance and its subpackages, internal/wire, and the packages of the hook handlers in the real txn.Registry registrations (field Handler of the M1-13 registrations, package derived from the function name). Any other package fails the test. A fixture package under internal/wire/testdata that references Counters must be reported (negative test).
  • Commit order: the row lock taken by Advance is held until commit; in READ COMMITTED a waiting transaction rereads the committed value (04 "Pattern with a counter row" items 1 to 4). Nothing in this module may run SELECT ... FOR UPDATE on the counters outside Advance.
  • instance row: id = 1 (CHECK (id = 1)), instance_id and sync_epoch UUIDv7 from kernel/ids, schema_version from goose, created_at default. Created by migrate up (03 instance). Concurrent migrate runs are safe by INSERT ... ON CONFLICT (id) DO NOTHING followed by the update.
  • Readiness message: /readyz names only the check (schema_requires_newer_server), no versions (16 "Two listeners"); the log line at start gives both versions.
  • Start check runs once at start (02 "Start" step 5 and 16 "Upgrade" item 4: the process keeps running but never becomes ready).

Tests (write them first)

Integration tests use pgtest and connect as bothy_app unless stated.

  • TestMigration_InstanceTablesMatchSpec: columns, types, nullability, defaults, checks and keys of both tables read from information_schema and pg_constraint equal the 03 definition.
  • TestMigration_CounterRowsCreated: three rows with value 0.
  • TestCounters_ConcurrentAdvanceGaplessInCommitOrder: many goroutines run units whose test sync hook calls Advance(event_seq, k) with random k and writes the range into a scratch table; a random sleep after Advance and random rollbacks; a reader polls the scratch table and at every read the visible ranges form one contiguous block from 1; at the end the block ends at the committed counter value.
  • TestCounters_RollbackLeavesNoGap.
  • TestCounters_PhaseGuard: Advance in work, Advance(event_seq) in audit, Advance(audit_seq) in sync, Advance(permission_version) in sync, Current in work all fail with kind internal; the allowed combinations succeed.
  • TestCounters_RejectsUnknownNameAndNonPositiveN.
  • TestCounters_ValueNeverDecreases: a direct UPDATE lowering a value (as bothy_owner) raises; bothy_app cannot INSERT or DELETE counter rows.
  • TestCountersOnlyUsedByPhaseHooks and its fixture case.
  • TestState_RecordMigrationCreatesAndKeepsIdentity: first call creates the row with UUIDv7 IDs; second call keeps instance_id, sync_epoch and created_at, updates the versions.
  • TestState_LoadMissingRow.
  • TestMigrateUp_RecordsSchemaAndMinServerVersion (command level): after migrate up the row has the goose version and MinServerVersion; with a database newer than the binary (simulate by applying an extra test migration) the row stays unchanged and the exit code is 0.
  • TestStart_MinServerVersionAboveOwn_NotReady: set min_server_version above the own version; start the server on port 0; /readyz returns 503 naming schema_requires_newer_server; /healthz returns 200.
  • TestStart_InstanceRowMissing_AbortsWithHint.
  • TestCompareSemVer: table including pre-release ordering from SemVer 2.0.0 section 11 examples and invalid input.
  • TestMinServerVersionNotAboveOwnVersion: MinServerVersion is valid SemVer and not greater than the build version.

Commands that must pass:

cd services/server && go test -race -count=1 ./internal/instance/... ./internal/wire/... ./cmd/...
make migrations-lint
make codegen-check
make arch
make check

Acceptance criteria

  • Concurrent increments are gapless and follow commit order, also with rollbacks.
  • Counters can only be advanced in their phase; the static check limits Counters to instance, wire and registered phase hooks (fixture proves it fails otherwise).
  • Counter values cannot decrease; bothy_app cannot insert or delete counter rows.
  • migrate up records instance_id, sync_epoch, schema_version, min_server_version and leaves a newer schema untouched.
  • A start with min_server_version above the own version does not become ready and /readyz names schema_requires_newer_server.
  • The setup state is available from Record.SetupCompleted() at start.
  • README and CHANGELOG updated; 02, 03, 18 and upgrade.md still match the build; make check green.

Definition of done

Code, tests and docs are complete: internal/instance/README.md current (status, ports, tables, rules), doc comment on every public symbol (purpose, parameters, return value, errors, example where useful), comments explain why. No new ADR (R-DM-05 and ADR 0009 cover the decision); rulings go into docs/DECISIONS.md. CHANGELOG entry under "Unreleased" in services/server/CHANGELOG.md. The migration has its final sequential number before the merge. make check green. Independent review (make review plus a code review over the diff) and the red team gate (docs/quality/red-team-gate.md) over the whole diff with verdict Urteil: JA recorded in the report; findings fixed with a test each and the delta re-checked. Merge locally into the base branch only if review and red team are green, then git worktree remove.

Report

Write to {{REPORT_PATH}}: first line Verdict: DONE or Verdict: BLOCKED <reason>; then the commits, the files with line counts, the test results with counts (including the red output of the first test run), what is not proven, and follow-ups that could not be done in scope with the reason. Then write {{DONE_PATH}} with exit=0 sha=<merge sha> or exit=1 <reason>.