M2-31: Conformance suite with the Fake-Agent¶

Status: ready | Depends on: M2-13, M2-15, M2-17, M2-18, M2-19, M2-20, M2-27, M2-29 | Milestone: M2

Goal¶

After this ticket the mandatory agent scenarios of 09 "Required scenarios" run data driven against the real server in every make check. Each test ID AG-01 to AG-32 has one scenario file under services/server/test/agents/scenarios/; a Go conductor in services/server/test/agents starts the real server in the process (shared harness of M1-71, own PostgreSQL 18 through pgtest, fake clock), acts for the people through the public API, checks the stored state, and drives one or more Fake-Agent processes. The Fake-Agent is the scriptable adapter of 09 "Test doubles": built on the Python SDK of M2-29 (module bothy_sdk.testing.fake_agent, the plan's sdk/python/testing), it connects as a named consumer of its agent, verifies invocation tokens with the SDK, keeps an inbox, acknowledges, reads the context, starts, patches, waits, resumes, renews and finishes runs, sends heartbeats, posts answers and cards, registers commands and reports sessions, and on command plays the faults the scenarios need: a wrong base_version, an expired or forged token, a wrong actual mode or tool class, a crash after accepting, a slow start. Only the public API lies between the two sides (ADR 0022 "Confirmation"). Why: M2-04 to M2-20 proved the adapter contract module by module with Go test clients; this suite proves it end to end with a real SDK on the other side, it is M2 exit criterion 1, and it is the conformance check that RK-09 promises for every adapter. Parts of a scenario that need a ticket of a later milestone (a message of origin import in AG-01, attachments and transcripts in AG-20, voice messages in AG-25) are held in a golden list of deferred parts with the ticket that brings them; nothing is skipped.

Read first¶

docs/PLAN.md: section 1 "Principles"; section 3 "Way of working per ticket"; section 5 "Test strategy" (rows "Integration server" and "Test doubles"); section 8 "M2 Agent platform, extensions and terminal" (Goal, Scope, Out of scope, Exit criteria 1, 5 and 10, Test strategy, Red team, Dependencies, Risks); the row M2-31 (this ticket), the rows of what exists (M2-04 to M2-20, M2-23, M2-29) and of who builds on it (M2-36, M2-37); the rows M2-24, M2-27, M3-07, M3-09, M7-27 and M8-03 (deferred parts and later users of the Fake-Agent); section 19 P-09, P-10, P-11, P-26.
Dependencies: the header names M2-13, M2-15, M2-17, M2-18, M2-19, M2-20, M2-27 and M2-29. Implied through them and therefore not in the header: M2-24, M2-23, M2-16, M2-14, M2-12, M2-11, M2-10, M2-09, M2-08, M2-07, M2-06, M2-05, M2-04, M2-02 and the M1 tickets M1-12, M1-13, M1-14, M1-16, M1-17, M1-28, M1-29, M1-31, M1-32, M1-44, M1-45, M1-51, M1-54, M1-55, M1-56, M1-60, M1-63, M1-64, M1-65, M1-66 and M1-71. Dependencies outside the M2 agent block: M1-71 (services/server/test/harness, BuildMatrixWorld) with M1-37 (the owner bootstrap that harness.Start uses), M2-27 and through it M2-24 (the two paths of origin app for AG-01 and Tracker in AG-20).
docs/spec/09-agents.md: 1 "Principles"; 2.1, 2.2 "Features", 2.4 "Tool class"; 3.1 to 3.7; 4.1 to 4.10 including 4.5.1 "Fresh token for a waiting run" and "Keys and rotation"; 5.1 "State machine", 5.2 "Reasons (runs.state_reason)", 5.3, 5.5; 6 "Trigger state (trigger_state)"; 7 "Follow-ups during a run"; 8 "Stop"; 9.4, 9.5; 10.1 to 10.5; 11.2 to 11.6; 12 "Auto-title"; 13; 14; 15 "Reliability"; 16.1 "Test doubles"; 16.2 "Required scenarios" (every row and the closing paragraph); 19; 21 "Configuration values of this chapter".
docs/spec/04-sync-protocol.md (core contract, read only): 2.2 "Envelope" (delivery fields redelivered and invocation_token, rule 3); 7.4 "REST variant"; 11.1 "Run document", 11.2 "From the adapter to the server"; 15.1 to 15.5 and 15.7 "Autocomplete"; 16.5 "Agent run with follow-up".
docs/spec/03-data-model.md (core contract, read only): runs, run_followups, message_components, interactions, messages (origin, trigger_state, trigger_state_reason, system_code), agent_sessions, agent_consumer_cursors, event_log, audit_events.
docs/spec/05-actions.md (core contract, read only): 4.10, 5 "Agent invocation agent.invoke", 6.
docs/spec/06-api.md: 2 "Basic rules", 5 "Errors", 7 "Idempotency", 15 "Contract tests and gate".
docs/spec/07-permissions.md: 4.1 "Decisions and reasons"; SR4; 10 "Examples" and 11.1 "Test matrix" (the world). docs/spec/10-backend-policy.md: 7 "Adapter check (I3)", 15 "Tests" (BR-06). docs/spec/11-extensions.md: 11.2 "Hand-written layer" (row testing), 11.3, 13 step 10. docs/spec/02-architecture.md: 6 "Code layout" (test/ and the component testsupport), 7.1, 12.3 "Job catalogue". docs/spec/23-quality.md: 1 "Principles", 3 "Component contract", "Documentation in code". docs/spec/19-security.md: SEC-LOG-2.
docs/api/openapi.yaml: the operations the scenarios call (getDiscovery, getJwks, rotateSigningKey, createGrant, revokeGrant, createChannel, putChannelBehavior, setAgentToolClass, installApp, createToken, createIncomingWebhook, postIncomingWebhook, createThread, getThread, setThreadTitle, closeThread, createMessage, createInteraction, updateComponent, respondToInteraction, listAgentSessions, bindAgentSession, reportAgentSession, sendHeartbeat, getConsumerState, pollAgentEvents, ackAgentEvents, getRun, patchRun, getRunContext, startRun, resumeRun, renewInvocationToken, awaitRunInput, finishRun, cancelRun, registerCommand, autocompleteCommand, answerAutocomplete); schemas RunStart, InvocationTokenRenew, InvocationTokenGrant, RunAwaitInput, RunFinish, RunCancel, Run, RunPatch, JsonPatchOperation, RunContext, RunContextMessage, Heartbeat, BillingEvidence, ToolClass, AgentSessionBind, AgentSessionReport, ChannelBehavior, IncomingWebhookCreate, MessageCreate, Message, EventEnvelope, EventPage, AgentInvocationPayload, AgentFollowupPayload, InteractionCreatedPayload, RunCancelRequestedPayload, CommandAutocompleteRequestedPayload, FieldError, Problem (x-problem-codes).
docs/api/asyncapi.yaml: messages event.agent.invocation, event.agent.followup, event.interaction.created, event.command.invoked, event.run.cancel_requested, event.run.state_changed, event.message.updated, event.command.autocomplete_requested; operations receive.ack, receive.heartbeat, receive.run.update, receive.command.autocomplete_result.
ADRs 0022 ("Confirmation": contract tests only through the public API of the running server), 0023 ("Confirmation": AG-08, AG-28, AG-31, AG-32), 0026 (generated core plus hand layer, FakeServer, shared scenarios), 0053 (no JSON null), 0054 ("Confirmation": AG-20, AG-21).
docs/DECISIONS.md: D-W-15, D-W-16, D-W-44, D-W-58, D-W-100; section 10 (rows D-M2-04-<n> to D-M2-20-<n>, D-M2-23-<n> and those of M2-29; format of new rows).
docs/security/threat-model.md: T-TK-05 (reuse of an invocation token), T-AI-07 (loops), T-AI-08 (personal subscription used by another person). docs/RISKS.md: RK-09, RK-13, RK-18. docs/OPEN-QUESTIONS.md: OQ-01 (strict I1, so no second person ever has trigger where a personal agent operates), OQ-17 (package names; nothing is published).
docs/operations/configuration.md: agent.ack_timeout, agent.ack_persist_interval, agent.cancel_grace, agent.presence_timeout, agent.heartbeat_persist_interval, agent.invocation_token_ttl, agent.jwks_max_age, agent.context_max_bytes, agent.context_message_max_bytes, agent.run_update_min_interval, auth.clock_leeway, run.orphan_timeout, run.queued_max_age, command.autocomplete_timeout, signing_key.invocation_rotation_interval, signing_key.cache_ttl, signing_key.rotation_check_interval, adapter.invocation_token_leeway, adapter.jwks_refresh_min_interval, sdk.http_timeout, sdk.retry_backoff_base, sdk.retry_backoff_cap, sdk.retry_max_attempts, http.request_timeout (all read, none new).
docs/quality/gates.md (rows test and coverage), docs/quality/red-team-gate.md, docs/quality/push-discipline.md.
M1 briefs whose interfaces you use: M1-12 (pgtest.Main, pgtest.New, one container per test binary), M1-14 (internal/platform/jobs: registry, Handler, PeriodicJob, roles), M1-31 (response validator in services/server/test/apitest), M1-64 (wstest), M1-71 (the shared harness: harness.Start, Options with Config and Clock, Server with PublicURL, APIBase, GatewayURL, Owner, AdminDB, NewUser, NewAgent, Client.Do, Client.StepUp, BuildMatrixWorld; "extend it only additively").
M2 briefs (all in docs/briefs/), for names and for the tests that recorded the red runs (Design note 14): M2-04-trigger-decision.md (AG-01 to AG-06 as decision tables), M2-05-runs-state-machine.md, M2-06-invocation-token.md (AG-22 at token level, the reference verifier, its note 7 on the run level), M2-07-agent-consumers.md (pollAgentEvents, ackAgentEvents, the ack save ticker and shutdown flush), M2-08-invocation-delivery.md (AG-07, AG-09, AG-27, job runs.queued_expiry), M2-09-run-lifecycle.md (AG-08, AG-10, AG-11, AG-14, AG-28 to AG-32, jobs runs.cancel_timeout and runs.stall_check), M2-10-run-document-live-box.md (AG-12, 409 conflict with reason run_not_active), M2-11-run-followups.md (AG-13), M2-12-agent-sessions.md (AG-17, AG-18, agentstest.InsertImportedSession), M2-13-run-context.md (AG-20, AG-21, the messaging test helper for messages of origin import), M2-15-components-interactions.md (AG-15, AG-16, AG-31 end to end), M2-16-slash-commands.md (AG-24), M2-17-command-autocomplete.md (AG-23), M2-18-auto-title.md (AG-19), M2-19-agent-messages.md (AG-01 for origin = agent, AG-26), M2-20-heartbeat-presence.md (sendHeartbeat), M2-23-generated-clients.md (layout of sdk/python, bothy_sdk, bothy_sdk.generated, tolerance), M2-24-apps-install.md and M2-27-webhooks-incoming.md (the two paths of origin app).
M2-29 (docs/briefs/M2-29-python-sdk.md): read its brief, its merged code and README under sdk/python (modules auth, gateway, invocation, runs, components, commands, outbox, retry, heartbeat, testing with the FakeServer, after 11 11.2) and its rows in docs/DECISIONS.md section 10.
Code on the base branch: services/server/test/harness, services/server/test/apitest, services/server/internal/platform/pg/pgtest, internal/platform/jobs, internal/platform/config, internal/wire, internal/runs (the test helper with M2-06's reference verifier), internal/agents/adapters/agentstest, the messaging test helper of M2-13, api/operations.yaml, architecture/modules.yaml; sdk/python (pyproject.toml, src/bothy_sdk/, tests/); tools/ci/components.py and tools/ci/tests/test_components.py.

Fixed requirements that apply¶

Owner's fixed build rules (apply verbatim):

No MVP. New from scratch and directly in the final state; stages are only the order.
Strictly modular: clear module and service boundaries, dependencies only through defined interfaces, no cross access.
Documentation in code: every public module, class, function and interface with a docstring/doc comment (purpose, parameters, return value, errors, example where useful); comments explain the why, not the what. API documentation is generated from code (OpenAPI from the routes, dartdoc, Python API docs in MkDocs).
Documentation in the repo: README per module/service, ADR for every architecture decision, CHANGELOG per release, keep the documentation site current. A change without matching documentation is not done.
Quality gates (SonarQube, linters, type checking, security scanners, test coverage) block the merge.

Rules for every brief:

Test first: write the failing test, run it and keep the red output for the report, then make it green, then clean up.
No en dash and no em dash in any file you write; the text-rules pre-commit hook rejects them. Use a hyphen or rephrase.
White label: examples and test data use example.com and the people Alex, Sam and Robin; no company, host or personal names, no paths under a home directory.
Limits, windows and thresholds only through named configuration keys listed in docs/operations/configuration.md; code comments and docs name the key, never the number.
Specification first: a change to behaviour, API, event, right or configuration updates docs/spec/ or docs/api/ in the same change. The core contract (docs/spec/00-glossary.md, 03-data-model.md, 04-sync-protocol.md, 05-actions.md) is not edited by this ticket.

Product requirements implemented here:

AG-01 to AG-32 of 09 16.2 run data driven against the real server in make check (M2 exit criterion 1; the plan row says AG-01 to AG-30, 09 16.2 also lists AG-31 and AG-32, and the chapter wins, PLAN 1 point 3); each scenario is one data file, and its expectations come literally from 09 16.2 and the chapters it names, never from server code.
The Fake-Agent is a scriptable adapter on the Python SDK with scenarios as data, and it can play faults: wrong base_version, expired token, wrong actual mode, crash after accepting (09 16.1, 11 11.2 row testing, ADR 0026).
The Fake-Agent uses only the public API, REST and gateway, with its own agent API token (ADR 0022 "Confirmation", 09 1 points 1 and 2).
The scenarios run against the real server with its own PostgreSQL instance (09 16.2 last paragraph); a missing Docker-compatible runtime or a missing uv fails the suite, never skips it (P-09); every listener binds 127.0.0.1 and port 0 (P-10).
No real agent runtime, no model call and no third-party service runs (09 16.1 "Real runtimes ... only behind an explicit switch"; PLAN 3 point 5).
Tokens and credentials never appear in scenario files, logs, test output, failure messages, event payloads or audit details (04 2.2 rule 3, SEC-LOG-2, T-TK-05).
Every scenario was red before its implementation; the proof is in the report of the implementing ticket, and this ticket's report collects it per AG ID (plan row M2-31).
A part of a scenario that needs a later ticket is recorded with that ticket in a golden list of deferred parts and is never silently left out (09 16.2 closing paragraph, P-09).

Scope¶

Paths are relative to the repository root. Names of M1-71, M2-04 to M2-20, M2-23 and M2-29 below are the expected ones; where the merged code names them differently, follow the code (Boundaries).

Create:

services/server/test/agents/ (component testsupport, Go package agents, only _test.go files, so the suite stays out of the product binary and the Semgrep egress rule's *_test.go exclusion covers the proxy): main_test.go (TestMain with pgtest.Main), conformance_test.go (TestConformance with one subtest per scenario file), scenario_test.go (types, strict loader, Validate), conductor_test.go (step execution, waits, keeps, parallel groups), checks_test.go (state checks through AdminDB), fakeagent_test.go (process manager and control client of the Fake-Agent), proxy_test.go (the fault proxy of Design note 9), world_test.go (world extensions of Design note 10), registry_test.go (coverage of 09 16.2 and the deferred golden list), and the unit tests of Tests below.
services/server/test/agents/scenarios/ag-01.yaml to ag-32.yaml: one file per test ID (Design note 3).
services/server/test/agents/testdata/deferred.golden (Design note 13).
services/server/test/agents/README.md (English, like the other READMEs under services/server): purpose, how to run one scenario, the scenario format, the Fake-Agent control protocol, the deferred parts, how a later ticket adds a scenario part.
sdk/python/src/bothy_sdk/testing/fake_agent.py (the Fake-Agent: class FakeAgent, entry point main() for python -m bothy_sdk.testing.fake_agent), sdk/python/src/bothy_sdk/testing/control.py (typed requests and replies of the control protocol), sdk/python/src/bothy_sdk/testing/token_faults.py (the token variants of AG-08). If M2-29 made testing a single module instead of a package, turn it into a package that re-exports its public names unchanged.
sdk/python/tests/testing/test_fake_agent.py, test_control.py, test_token_faults.py: tests of the Fake-Agent against M2-29's FakeServer (no real server, no listener other than the FakeServer's on 127.0.0.1 port 0).

Change:

services/server/test/harness (M1-71, additive only; Design note 8): Options.ManualJobs, Options.PublicOrigin, (*Server).ListenURL, (*Server).RunDueJobs, (*Server).Restart, (*Client).AgentCredentials, with unit or integration tests next to them and the harness README.
services/server/internal/messaging/adapters/messagingtest: only the fixture helper of Design note 10, and only if the base branch has none.
sdk/python/src/bothy_sdk/: only the hook of Design note 5 f (an injectable clock for token expiry and JWKS cache age), with a test, if M2-29 did not provide it.
tools/ci/components.py: the entry server of GATE_COMPONENTS also owns the prefix sdk/python/ (Design note 15); tools/ci/tests/test_components.py: a case that sdk/python/src/bothy_sdk/testing/fake_agent.py selects server and sdk-python.
docs/spec/09-agents.md (English, like the chapter, not core contract): 16.1 row "Fake agent" (control by the suite over JSON lines, module bothy_sdk.testing.fake_agent, only public API); 16.2 closing paragraph (the suite in services/server/test/agents, one file per ID, the deferred parts with their tickets).
docs/spec/11-extensions.md 11.2 row testing: "in-process FakeServer with fault injection, fake agent for the conformance suite, scenario helpers, recorded fixtures".
docs/spec/02-architecture.md 6 "Code layout": agents/ in the list of test/ ("conformance suite of the agents with fake agent, M2-31").
docs/adr/0022-agents-and-plugins-outside-the-server-process.md "Confirmation": the contract-test bullet names services/server/test/agents (TestConformance).
docs/quality/gates.md row test: the conformance suite runs inside go test of services/server and starts the Fake-Agent from sdk/python through uv.
sdk/python/README.md (section "Test doubles": the Fake-Agent and its control protocol), services/server/test/harness/README.md (the additions); services/server/CHANGELOG.md, sdk/python/CHANGELOG.md and the root CHANGELOG.md (CI path filter) under "Unreleased"; docs/DECISIONS.md section 10 rows D-M2-31-<n> (Design note 16).

This ticket creates no migration, no operation, event, action, job, metric or configuration key, adds no Go or Python dependency, and creates no new top-level make target.

Out of scope¶

Every server behaviour the scenarios check: M2-04 to M2-20 built it. A scenario that fails because the server deviates from the specification is not fixed here (Boundaries).
The SDK hand layer itself, the FakeServer and the SDK's own scenarios against it (AG-08 adapter side, AG-09, AG-12, AG-28 per M2-29's row), the adapter-side conformance of other adapters (M2-29, M2-30, M2-33, M2-34, M2-35).
AG-01 with origin import (M8-03, the first path that writes such messages), the attachment and transcript parts of AG-20 (M3-07, M3-09) and AG-25 (M3-09): deferred parts (Design note 13; plan gap reported).
Load on the agent path (perf.invocation_delivery_p99, perf.run_patch_fanout_p99): M2-36. The guide check against a fresh server: M2-37. The demo instance that lets the Fake-Agent produce runs (P-26): M7-27.
Red-team scenarios beyond the AG IDs (personal subscription over every path, confused deputy): covered by the tests of M2-08, M2-11, M2-14 to M2-17 and M2-19.
The TypeScript Fake-Agent and the TypeScript run of the same scenarios (M2-30).
docs/PLAN.md (the gaps go into the report), changes to the core contract.

Boundaries¶

Work in a dedicated git worktree on the branch feat/M2-31-agent-conformance-suite; one writer per worktree.
Conventional Commits with the component as scope (for example test(agents): ..., feat(sdk-python): ..., test(harness): ..., ci(components): ...) and a DCO sign-off on every commit (git commit -s); squash into meaningful commits before the merge (docs/quality/push-discipline.md).
Never push to main, never tag and never create remote repositories; the only remote steps are the branch push and the pull request of AGENTS.md. Never publish a package (OQ-17).
Do not modify files outside the scope above, except the CHANGELOG, README and docs/DECISIONS.md entries named in this brief.
Follow the host rules supplied by the launcher (ports, machine rules).
Tests bind only 127.0.0.1 and port 0; PostgreSQL runs only as a throwaway container on a random port through pgtest (P-09, P-10). Without a Docker-compatible runtime or without uv on PATH the suite fails with a message that names what is missing; it never skips.
Stop only processes you started, by PID: every Fake-Agent process is killed in t.Cleanup and its exit is awaited. Temporary directories only via mktemp -d or t.TempDir(), removed at the end.
Never read or print secrets. No network calls except package downloads for pinned tools, Go modules and the locked Python packages of sdk/python.
Never lower a threshold, disable a gate or delete a test. The go test line of the root Makefile is not changed (no -timeout, no tags, no -short).
If the core contract lacks something this ticket needs, stop with Verdict: BLOCKED <reason> instead of building a deviation (P-11).
This brief was written before milestone M1 closed (D-W-100). Where the merged code of an M1 ticket names a package, type, function, file or make target differently from this brief, follow the code and list each difference in the report; where it behaves differently from the specification, stop with Verdict: BLOCKED <reason>.
The same holds for the merged code of M2-04 to M2-20, M2-23, M2-24, M2-27 and M2-29: follow its names and list each difference. A scenario that is red because the server or the SDK behaves differently from 09 16.2 or the chapter it names is not weakened and not worked around: stop with Verdict: BLOCKED <ticket> deviates: <AG ID> <expected> <observed>.
The Fake-Agent uses only the public API. Only the conductor reads the database (AdminDB, assertions and the snapshot of Design note 8), runs jobs and writes the fixtures of Design note 10 that have no API path; it never writes state that an operation can write.
No agent runtime, no model and no third-party service: the Fake-Agent decides everything from its script. Time only through the harness clock (kernel/clock); the conductor moves it between steps and never sleeps for a window of the specification.
Tokens are passed only in memory (control pipe, conductor variables); the conductor's failure messages, the captured stderr of the Fake-Agent and the scenario files never contain a token, a signature, an API token or a DPoP key (print the first hex characters of the SHA-256 of the jti when a token must be named).

Design notes¶

Start check. Run make check on the fresh worktree; if it is red, stop with Verdict: BLOCKED base branch red and the failing gate. Then confirm on the base branch: services/server/test/harness with Start and BuildMatrixWorld (M1-71); the operations startRun, resumeRun, renewInvocationToken, awaitRunInput, finishRun, cancelRun, patchRun, getRunContext, createInteraction, bindAgentSession, reportAgentSession, autocompleteCommand, answerAutocomplete, registerCommand, setThreadTitle, sendHeartbeat, installApp and postIncomingWebhook under enabled in services/server/api/operations.yaml; sdk/python/src/bothy_sdk with the hand-layer modules of 11 11.2 and the FakeServer in bothy_sdk.testing (M2-29). A missing one ends with Verdict: BLOCKED dependency <ticket> missing.
Architecture. Ruling for docs/DECISIONS.md: the conductor is Go and the Fake-Agent is Python. The Go side owns what only it can own: the real server in the process with its fake clock, pgtest, the matrix world of M1-71, the jobs, the database checks and the response validator of M1-31. The Python side is what 09 16.1 demands: an adapter on the SDK. One Fake-Agent process per agent and consumer of a scenario, started with uv run --project <repository root>/sdk/python --locked --quiet python -m bothy_sdk.testing.fake_agent; the repository root is found by walking up from the test directory to the directory that holds sdk/python/pyproject.toml. If M2-29 already built a Go-side runner that starts the real server for its own real-server tests, extend it instead of building a second one and list it in the report. If a linter flags the start of the subprocess (the program is uv from PATH, the arguments are constants and repository paths), add a suppression at that line with this reason, never a global exclusion.
Scenario files (scenarios/ag-NN.yaml; parsed with the YAML library already in services/server/go.mod, strict decoding with known fields only; if the module has no YAML library, the files are JSON with the same structure and the extension .json, and no dependency is added). The format is defined by the Go types in scenario_test.go and their Validate method and documented in the README; there is no separate schema file. Top-level keys:
- id (exactly the file's ID), title (the English row text of 09 16.2), sources (chapter references such as 09 16.2, 04 15.5), world (matrix), config (dotted keys of docs/operations/configuration.md with test values; every key the scenario advances the clock by must be set here), setup (steps run before the first case), agents (name to {agent, consumer, transport, runtime, launch}; transport is rest or gateway; launch is start (default) or later; runtime gives the actual billing mode, billing evidence, loaded tool class, reported capabilities and the adapter options adapter.invocation_token_leeway and adapter.jwks_refresh_min_interval), cases (named lists of steps; each case runs on its own server and world unless the scenario sets shared_world: true), matrix (only AG-02 and AG-12: rows of values substituted into the steps, each row with its literal expected outcome), deferred (parts with ticket and reason, Design note 13).
- Step kinds, exactly one per step: as: <person> (or the subjects tracker and webhook of Design note 10) with call: <operationId> and with (a call through that subject's client); agent: <name> with do: <op> and with (a Fake-Agent op, Design note 5); server: one of advance_clock (by: <key>, moves by that key's configured value plus one smallest tick of the fake clock), run_jobs (kind: <job kind>), snapshot_cursors (keep: <name>, the stored rows of agent_consumer_cursors), restart (optional restore_cursors: <name>), proxy (drop_next: <operationId>); check: one of run, runs, message, followup, component, interaction, session, thread, events, audit, consumer (Design note 4); parallel: a list of steps released together behind a barrier; repeat: {times, steps}; await: <name> with an expect, which waits for a human call that an earlier step started with async: <name> (a call that blocks inside the server, such as autocompleteCommand).
- expect on every call and op: status, code, reason, errors (list of {code, pointer}), none (no event of a type arrived), one_of (alternatives), and op-specific fields (event, verdict, redelivered, trigger_kind, session_mode); keep binds names to IDs from answers (run: r1, message: m1, token: t2) for later steps ($r1). Names of people, agents, spaces, channels and threads are the world's names.
- Expectations are literal data. AG-02 lists all its rows with the expected outcome written out from 09 3.4; nothing in the suite calls Decide of M2-04 or any other server function to compute an expectation.
Example (illustrative, AG-14):
```
id: AG-14
title: Stop without an adapter answer
sources: ["09 16.2", "09 8", "04 15.5"]
world: matrix
config: {agent.cancel_grace: <test value>}
agents:
  lotse: {agent: Lotse, consumer: default, transport: rest}
cases:
  main:
    - as: alex
      call: createMessage
      with: {thread: support-allgemein-1, content: "@lotse please check the export"}
      expect: {status: 201}
      keep: {message: m1}
    - agent: lotse
      do: receive
      expect: {event: agent.invocation, verdict: valid}
      keep: {run: r1}
    - agent: lotse
      do: start
      with: {run: $r1}
      expect: {status: 200}
    - as: alex
      call: cancelRun
      with: {run: $r1}
      expect: {status: 202}
    - server: advance_clock
      by: agent.cancel_grace
    - server: run_jobs
      kind: runs.cancel_timeout
    - check: run
      run: $r1
      expect: {state: cancelled, state_reason: cancel_timeout}
    - agent: lotse
      do: patch
      with: {run: $r1, ops: [{op: replace, path: /status_line, value: "still here"}]}
      expect: {status: 409, code: conflict, reason: run_not_active}
```
Conductor. Executes the steps of a case in order on a fresh harness.Start with Options{Config: scenario config, Clock: a fake clock, ManualJobs: true, PublicOrigin: the origin of the case's fault proxy} (Design note 9) and the world of Design note 10.
- At the start of each case it starts the Fake-Agent process of every entry in agents and runs hello and connect, so every consumer's cursor exists before the first trigger (04 15.1: the cursor starts at head_seq on the first connection). An entry with launch: later (a restarted adapter in AG-10, a second process) is started by a step agent: <name> with do: launch, which runs hello and connect.
- Human calls go through the harness client of that person (DPoP, Idempotency-Key, step-up for the Owner where the operation needs it); every response is validated with the response validator of services/server/test/apitest (M1-31) and must contain no JSON null.
- Checks read the stored state through AdminDB() with read-only SQL: runs (state, state_reason, version, trigger_kind, triggered_by, agent_session_id, billing_mode_used, and started_jti, invocation_jti, renewed_from_jti compared only by their hash prefix), messages (origin, kind, system_code, trigger_state, trigger_state_reason), run_followups, message_components, interactions, agent_sessions, threads (title, title_source, parent_message_id, operator_agent_id), event_log (type, audience_principal_id, resource_kind, resource_id, order by seq, count), audit_events (action, decision, deny_reason, keys of details), agent_consumer_cursors. Reads go through AdminDB, also for history and audit rows, although the history and audit reads (M1-57, M1-50) are in this ticket's closure; every read names its table and column as 03 does.
- Asynchronous effects (a delivery over the gateway, a reconnect after a restart) are awaited by polling the condition until it holds or the test's deadline (t.Deadline()) is reached; no other wait exists, and no step waits for a window of the specification in real time.
- A failed expectation names the scenario, case, step index, the expected and the observed value (codes, states, counts), never a token or a body that could contain one.
- Cases run with t.Parallel(); pgtest starts one container per test binary and clones a database per case. The package must finish within the default timeout of go test, because the Makefile line stays as it is; if it does not, first let the cases of one file share their world (shared_world: true) where they touch different threads, and report the wall time either way.

Fake-Agent (bothy_sdk.testing.fake_agent; public API of the module documented with google-style docstrings, rendered by M2-29's MkDocs reference). a. Control protocol: JSON lines on stdin (requests {id, op, args, now}) and stdout (replies {id, ok, result} or {id, ok: false, error: {kind, detail}}); the first request is hello with base_url, gateway_url, agent_id, consumer, transport, runtime, the adapter options and the credentials (Design note 6). Every request carries now, the harness clock as RFC 3339 UTC with milliseconds; the Fake-Agent sets its SDK clock to it before it acts. Logs go to stderr at level WARNING and never contain a token, a credential or message content; stdout carries only replies. b. Ops (each through the SDK hand layer; a request the hand layer would never send, such as a forged, empty or absent token, a stale base_version or a start past the adapter's own checks, goes through the generated core bothy_sdk.generated directly):

Op	Effect	Result fields
`connect`	register the consumer before the first trigger of a case (REST: a first `pollAgentEvents`, which creates the cursor and returns an empty page; gateway: `identify`), so its cursor starts at `head_seq` (04 15.1)	`head_seq`
`receive`	consume until an event of the requested type arrives or the consumer reached `head_seq` (REST: `pollAgentEvents` pages; gateway: dispatch frames); verify its token with the SDK (`invocation`), store event and token in the inbox (newest token wins for the same event, 09 4.3) and acknowledge, unless `accept: false` or the verdict forbids it (09 4.5: unknown `kid` after one reload is neither accepted nor acknowledged); `hold: true` keeps the event unverified and unacknowledged until a later `verify` of it	`event`, `seq`, `redelivered`, `run_id`, `thread_id`, `message_id`, `interaction_id`, `command_id`, `trigger_kind`, `session_mode`, `verdict`, `jti_hash`, or `none`
`start`, `resume`	`startRun` or `resumeRun` with the newest inbox token of the run, or the token named in `token` (`inbox`, a kept name, `empty`, `absent`, `forged:<variant>`), with `billing_mode_used`, `tool_class_used`, `billing_evidence` and `model` from the runtime or overridden	`status`, `code`, `reason`, `run_state`, `replay`
`renew`	`renewInvocationToken` with the named token; on success the new token goes into the inbox; option `retry: false` sends this one call through a client whose `SdkSettings.retry_max_attempts` is 1, so a dropped answer is reported as `error {kind: transport}`	`status`, `code`, `reason`, `jti_hash`
`context`	`getRunContext`; per element `message_id`, `role`, `trust`, `truncated`, byte length; the SDK helper composes the prompt with marked blocks (ADR 0054), and the Fake-Agent reports for every element with `trust` other than `trigger` and `self` whether its text occurs only inside a marked block	`trigger_complete`, `elements`, `truncated_before_seq`
`patch`	`patchRun` (REST) or op `run.update` (gateway) with `ops`; `base_version: stale` sends the version before the last stored change; the SDK's rebase after `409 base_version_conflict` runs unless `rebase: false`	`status`, `code`, `errors`, `current_version`
`await_input`, `finish`	`awaitRunInput`; `finishRun` with `state`, `reason`, `summary`	`status`, `code`, `reason`
`heartbeat`	`sendHeartbeat` with `active_runs` (default: the runs it holds), `capabilities`, reported mode, evidence and tool class from the runtime	`status`
`post`	`createMessage` or `createThread`, plain or as a card (the SDK `components` builders, the approval card of 09 11.5)	`status`, `code`, `reason`, `message_id`, `component_ids`
`respond`, `update_component`, `set_title`, `report_session`, `register_commands`	`respondToInteraction`, `updateComponent`, `setThreadTitle` (with the `ETag` of `getThread`), `reportAgentSession` (with `If-Match`), `registerCommand` through the SDK's declarative command list	`status`, `code`, `reason`
`autocomplete_mode`	`answer` with given choices, or `ignore` (receive `command.autocomplete_requested`, report it, never answer)	`requested`
`verify`	SDK verification of a named token without any request; for a held event, acceptance and acknowledgement then follow the verdict as in `receive`	`verdict`
`export_token`, `import_token`	hand a token of this agent's inbox to the conductor, or take one into the inbox (AG-08 foreign `aud`, AG-32 lost answer)	`jti_hash` (and `token` only in `export_token`)
`crash`	exit at once with a non-zero status, no `finishRun`, no flush	none

c. Verdicts of verify and receive, mirroring M2-06's reference verifier: valid, expired, invalid, unknown_key, billing_mode_mismatch, tool_class_mismatch, replay. d. Well-behaved default (09 4.9): after receive with verdict invalid, billing_mode_mismatch or tool_class_mismatch the Fake-Agent starts nothing and calls finishRun with failed and the matching reason (invalid_invocation_token, billing_mode_mismatch, tool_class_mismatch); a step with skip_own_checks: true sends the start anyway through the generated core (AG-11, AG-30 server parts). e. Token variants of forged:<variant> (token_faults.py): bad_signature (one byte of the signature changed), alg_hs256, alg_none (header re-encoded), typ_access (header typ of an access token), built from the run's current token; empty is an empty string; absent leaves the member out. f. SDK hook (Scope, Change): if the merged SDK has none, add a keyword argument clock (a callable returning an aware datetime, default the system clock) to the invocation verifier and its JWKS cache, used for expiry, iat and max-age, documented as a test hook; the default keeps M2-29's behaviour.

Credentials. The conductor reads the agent's API token, and its DPoP private key when the harness bound the token to one, through (*Client).AgentCredentials() and sends them only in hello on stdin; never in argv or the environment of the process. Ruling for docs/DECISIONS.md: credentials of a test double travel only over its stdin.
Time and jobs. One fake clock per case (the harness clock). The conductor advances it only between steps, after the previous step's effects were observed; the Fake-Agent follows through now. Jobs never run by themselves: Options.ManualJobs keeps the job runtime from working jobs, and run_jobs calls (*Server).RunDueJobs(kind), which runs every due job of that kind (scheduled at or before the harness clock) through its registered handler in id order, or a periodic kind (runs.stall_check, instance.key_rotation) once. Ruling for docs/DECISIONS.md: jobs in the suite run only on request, because River's own scheduling follows the wall clock while the server follows the fake clock. Kinds used: runs.cancel_timeout, runs.stall_check (M2-09), runs.queued_expiry (M2-08), instance.key_rotation (M1-32).
Harness additions (M1-71, additive; each with a doc comment and a test):
- Options.ManualJobs bool and func (s *Server) RunDueJobs(tb testing.TB, kind string) int (Design note 7).
- Options.PublicOrigin string (when set: server.public_url, and the origin of PublicURL, APIBase, GatewayURL and of every harness client) and func (s *Server) ListenURL() string (the actual public listener).
- func (s *Server) SnapshotCursors(tb testing.TB) CursorSnapshot (the stored rows of agent_consumer_cursors) and func (s *Server) Restart(tb testing.TB, opts RestartOptions) with RestartOptions{RestoreCursors *CursorSnapshot}: stops the wire graph gracefully and starts it again on the same database, secrets, configuration and clock, with new listeners on 127.0.0.1 port 0; ListenURL() changes; PublicURL, APIBase and GatewayURL stay Options.PublicOrigin when it is set, otherwise they and server.public_url follow the new listener. With RestoreCursors it writes the snapshot back between stop and start, which leaves the state of a process that died before the acknowledgements made after the snapshot were stored (a gateway acknowledgement is held in memory until the save ticker of agent.ack_persist_interval or the shutdown stores it, M2-07 note 9; a REST acknowledgement is lost only when the process dies before its transaction commits). Ruling for docs/DECISIONS.md: crash emulation by cursor snapshot, because an in-process server cannot be killed without killing the test.
- func (c *Client) AgentCredentials() AgentCredentials with AgentCredentials{Token string; DPoPPrivateJWK []byte} for clients made by NewAgent, documented as test-only and never logged.
Fault proxy (proxy_test.go). Every Fake-Agent talks to the server through one reverse proxy per case on 127.0.0.1 port 0 (REST and the WebSocket upgrade of the gateway), whose upstream transport is the loopback client of M1-22 (egress.NewLoopback) with the case's value of http.request_timeout. It keeps the Fake-Agent's base URL stable across restart (it switches its upstream and closes open WebSockets, so the SDK reconnects) and serves proxy: {drop_next: <operationId>}: the next matching request is forwarded, the server's response is read completely and kept in memory for the conductor (keep: {token: t2} takes the token of an InvocationTokenGrant), and the client connection is closed without an answer. This is the "fake proxy" of AG-32. Before harness.Start the conductor opens the proxy listener on 127.0.0.1:0 and starts the server with Options.PublicOrigin set to the proxy origin, which the harness also sets as server.public_url, because the server builds the DPoP htu from server.public_url (08 7.3, M1-18) and every client signs the URL it calls. Every client of the case (the people through the harness, tracker, webhook and every Fake-Agent, REST and gateway) goes through the proxy; its upstream is (*Server).ListenURL(). restart keeps server.public_url; the proxy switches only its upstream.
World. harness.BuildMatrixWorld (07 10 and 11.1: Support with Allgemein and Eskalation, Marketing with Kampagnen, Engineering with Projekt-A and Werkzeuge, Research with Notizen; Alex, Sam, Robin and the Owner; Lotse, Texter, Runner, Assistent; group Leser; grants G-1 to G-5), extended in world_test.go through public operations only: the tool classes of 07 10 with setAgentToolClass where the merged world left them at their default (Lotse chat, Texter scoped, Runner host, Assistent scoped); per scenario the channel switches with putChannelBehavior, extra grants with createGrant, channels with createChannel; for AG-01 and AG-20 the app Tracker through installApp with scope messages:write, its grant G-6 (write on Support/Eskalation) and its token through createToken; for AG-01 an incoming webhook of format native on Support/Eskalation through createIncomingWebhook. Tracker acts as the subject tracker through a plain HTTP client with its app token, and the incoming webhook as the subject webhook through postIncomingWebhook with the token the world kept in memory. Fixtures without an API path are written by the module test helpers on the harness database and only those: imported session rows through agentstest.InsertImportedSession (AG-18, M2-12) and messages of origin import in a thread through the messaging test helper that M2-13 used for them (AG-20); if the base branch has no such helper, add InsertImportedMessage(ctx, ImportedMessage{ThreadID, AuthorID ids.ID; Content string}) to the test-helper package of messaging (internal/messaging/adapters/messagingtest, 02 6), which stores the message through the append path that createMessage uses (thread sequence through the chat port) with origin = import and publishes no MessagePosted, with a test. Ruling for docs/DECISIONS.md: fixtures are allowed only for state that no operation can write before M8-03.
Transports. REST (pollAgentEvents, ackAgentEvents) is the default, because an empty poll up to head_seq proves that nothing was delivered without a timeout; every expectation none uses REST. The gateway is used where the scenario needs it: AG-12 (both transports), AG-23 (ephemeral command.autocomplete_requested reaches only a live gateway connection, M2-17). Ruling for docs/DECISIONS.md.
Rules for single scenarios. a. AG-02: three switches of 09 3.3 that decide a candidate (require_mention, thread_require_mention, free_response) in all eight combinations, times main thread and thread, times no mention and a mention of the operator: 32 rows, each in its own channel of Support created by the Owner with auto_thread = false (so the run stays in the thread of the message; auto_thread is AG-05's). A thread row's thread is created by Lotse's Fake-Agent with createThread (an agent message never triggers, AG-01), then Alex writes. The expected outcome per row is written out from 09 3.4: with the operator mentioned always a run; without mention a run exactly when free_response, or main thread and not require_mention, or thread and not thread_require_mention; otherwise no run and no trigger_state. b. AG-08 "missing" is the empty token (403 invocation_token_invalid with reason invalid, M2-09 note 3e); the absent member is a schema error (400 validation_failed) and is asserted as such. Ruling for docs/DECISIONS.md (spec gap reported). c. AG-22 emergency: the scenario follows 09 4.5 "Keys and rotation" point 3: the Fake-Agent takes the delivery with hold: true; the Owner calls rotateSigningKey with mode = emergency; the clock passes agent.jwks_max_age, so any JWKS the Fake-Agent cached has expired; the following verify fetches the JWKS, the token of the withdrawn key fails with unknown_key (the key check comes before the expiry check, 09 4.5 points 3 and 5), the event is neither accepted nor acknowledged, and once the clock has also passed agent.ack_timeout it comes again with a token of the new key. The case of an adapter whose still-valid JWKS cache accepted a token of the withdrawn key before the rotation fails its run at startRun; it contradicts "costs ... no run" and is reported as a spec gap (first found by M2-06), not asserted. d. AG-28: the slow start is the conductor holding the Fake-Agent between acknowledgement and context (09 16.2 "fake agent delays run.context.read"); the conductor takes snapshot_cursors after Alex's message and before the Fake-Agent's receive acknowledges T1's event, and the later restart restores that snapshot, which is the state a restart leaves when the acknowledgement was not stored (on the gateway it waits for the save ticker, M2-07 note 9); after the restart the next poll delivers the same event (same event id and seq) again, without redelivered, because the window of the new process is fresh (M2-07 note 5).
Deferred parts. testdata/deferred.golden holds exactly these lines, and registry_test.go compares the deferred entries of all scenario files with it: AG-01 origin=import M8-03 (no path writes such a message before M8-03, and a fixture row would not pass the acceptance), AG-20 attachment M3-07, AG-20 transcript M3-09, AG-25 voice M3-09 (the file pipeline with scan and the transcript path are M3 tickets with their own test infrastructure). The ticket named in a line adds its part to the scenario file and removes the line in the same change. Ruling for docs/DECISIONS.md; plan gap reported (M2 exit criterion 1 names AG-01 to AG-30 for M2, and M3-07 and M8-03 do not depend on M2-31; M3-09 does since D-W-108).
Red runs. The plan row requires that every scenario was red before its implementation, proven in the report of the implementing ticket. This ticket: a. writes each scenario file before the conductor and the Fake-Agent support its steps, runs it, and keeps the red output; b. proves that every kind of expectation bites (status, code, reason, errors, event, verdict, none, every check kind): the unit tests of Tests feed a wrong expected value and assert the step fails; c. reports a table with one row per AG ID: the implementing ticket and its test that recorded the red run (AG-01 M2-04 TestDecide_AG01_NonHumanOriginNeverTriggers and M2-19 TestAG01_AgentMessagesNeverTrigger; AG-02 to AG-06 M2-04 TestDecide_AG02_SwitchMatrix to TestDecide_AG06_ActiveRunOtherAgent; AG-07 M2-08 TestDelivery_AG07_AZM51_RevokedBeforeDelivery; AG-08 M2-09 TestStartRun_TokenInvalid_AG08; AG-09 M2-08 TestDelivery_AG09_TwoAdapterProcesses; AG-10 M2-09 TestStallCheck_Interrupted_AG10; AG-11 M2-09 TestFinishRun_AdapterReportsMismatch_AG11; AG-12 M2-10 TestAG12PatchStateRejected, TestAG12StaleBaseVersion; AG-13 M2-11 TestAG13_Steer and siblings; AG-14 M2-09 TestCancelRun_Running_AG14; AG-15, AG-16 M2-15 TestAG_15_ConcurrentClicksOneWins, TestAG_16_AlwaysAllowNonOwner403; AG-17, AG-18 M2-12; AG-19 M2-18 TestAG19_OperatorAgentTitleAfterRun; AG-20, AG-21 M2-13; AG-22 M2-06; AG-23 M2-17; AG-24 M2-16 TestAG24_ReservedNamesRefused; AG-26 M2-19 TestAG26_AgentWithoutWrite; AG-27 M2-08 TestQueuedExpiry_AG27; AG-28 to AG-32 M2-09 and M2-15 TestAG_31_InteractionResumesWaitingRun; AG-25 M3-09), with the merged test names, and cites the red output where the launcher made the earlier reports available; otherwise it names the gap under "What is not proven".
CI path filter. The suite runs inside the gate component server but executes the code of sdk/python; a change only under sdk/python/ must therefore run the server gates too. Ruling for docs/DECISIONS.md: GATE_COMPONENTS["server"] owns sdk/python/ in addition to services/.
Rulings for docs/DECISIONS.md (section 10, one row D-M2-31-<n> each, with date, reason and place): architecture (note 2); scenario format with literal expectations (note 3); state checks through AdminDB and response validation (note 4); the Fake-Agent ops, verdicts, well-behaved default and the SDK clock hook (note 5); credentials over stdin (note 6); jobs on request (note 7); crash emulation by cursor snapshot (note 8); the fault proxy (note 9); fixtures only without API path (note 10); REST by default (note 11); AG-02 rows, AG-08 "missing", AG-22 emergency (note 12); deferred parts (note 13); red-run proof (note 14); CI path filter (note 15); AG-31 and AG-32 in the suite (requirement 11).

Tests (write them first)¶

Unit, Go (services/server/test/agents, no server, no container):

TestScenarioFiles_Load: every file under scenarios/ loads with strict decoding, its id equals the file name, every step has exactly one kind, every call names an operationId of docs/api/openapi.yaml, every do names an op of Design note 5 or launch, every advance_clock key is set in config and listed in docs/operations/configuration.md, every run_jobs kind is one of Design note 7, every call and op has an expect, every case ends with at least one check, and a file without cases is accepted only when every part of its ID is in deferred; a file with an unknown key, a duplicate id or a missing expect is rejected (table of broken inline documents).
TestScenarioRegistry_CoversMandatoryScenarios: the IDs of the table in docs/spec/09-agents.md 16.2 equal the set of scenario IDs (AG-01 to AG-32), none missing, none extra.
TestScenarioRegistry_DeferredGolden: the deferred entries of all files equal testdata/deferred.golden line for line; each line names a ticket ID that exists in docs/PLAN.md.
TestScenarioFiles_NoSecretsAndWhiteLabel: no compact JWS (three base64url parts), no API token prefix of ADR 0061, no DPoP key, only example.com addresses and the world's names.
TestExpectations_Bite (Design note 14 b): for each expectation kind and each check kind, a step with a wrong expected value against a recorded reply or a fake AdminDB result fails with a message that names step, expected and observed value and contains no token.
TestFaultProxy_DropsAnswerAfterUpstream: the upstream handler runs and commits, the client gets a closed connection, the kept body equals the upstream answer; TestFaultProxy_SwitchesUpstreamAndClosesSockets; TestFaultProxy_PassesWebSocketUpgrade.
TestFakeAgentProcess_CredentialsOnlyOnStdin (the process's argv and environment contain no credential) and TestFakeAgentProcess_KilledOnCleanup (the PID is gone after cleanup).

Unit and integration, harness (services/server/test/harness): TestRunDueJobs_RunsOnlyDueJobsOfKind, TestRunDueJobs_PeriodicKindRunsOnce, TestRestart_SameDatabaseNewListeners, TestRestart_DPoPAgentRequestThroughProxy (an agent request with a DPoP proof through a proxy on PublicOrigin gives 200 before and after Restart), TestRestart_RestoreCursors (an acknowledgement made after the snapshot is lost, one stored before it stays), TestAgentCredentials_OnlyForAgents.

Unit, Python (sdk/python/tests/testing/, against M2-29's FakeServer, run by make test; these tests alone keep sdk/python above its coverage threshold of M2-23, because the Go suite's use of the module is not measured):

test_control.py: request and reply round trip; an unknown op, a missing hello and a malformed line give an error reply and the process keeps running; replies carry no token except export_token.
test_fake_agent.py: receive accepts a valid invocation, stores it and acknowledges only afterwards; a redelivered event replaces the stored token and is not processed twice; an unknown kid after one reload is neither accepted nor acknowledged; the well-behaved default finishes failed with the matching reason for invalid, billing_mode_mismatch and tool_class_mismatch; skip_own_checks sends the start anyway through the generated core; patch with base_version: stale reports the first answer and rebases; heartbeat lists held runs; a renew with retry: false whose answer the FakeServer drops (Fault.disconnect()) replies error {kind: transport} after exactly one request; crash exits non-zero without a request; the clock follows now for expiry and JWKS max-age.
test_token_faults.py: every variant of Design note 5 e is rejected by the SDK verifier with invalid; the original token is unchanged.
test_no_secrets_in_logs: with a canary token and a canary API token, captured stderr and replies (other than export_token) never contain them.

Conformance (TestConformance/<ID>, PostgreSQL 18 via pgtest, real modules through internal/wire via the harness, HTTP and gateway through the real chain behind the fault proxy, Fake-Agent processes, fake clock). World of Design note 10; "Lotse runs" means: Alex mentions @lotse in a thread of Support/Allgemein, Lotse's Fake-Agent receives and starts. Each file asserts:

AG-01 (origin agent, app, import, system in a free channel: no run): Support/Eskalation with free_response = true; Lotse's Fake-Agent posts without and with @lotse; Tracker posts with its app token mentioning @lotse; postIncomingWebhook posts a native body naming @lotse; the Owner closes a thread and Lotse answers in it, which writes the thread.reopened line; the stored messages have origin agent (Lotse), app (token), app (webhook), system (the reopen line) and agent (the answer), none has a trigger_state, and no runs row and no agent.invocation appear; control at the end: Alex writes without mention, one run of Lotse queued. Deferred: origin=import.
AG-02 (switch matrix): the 32 rows of Design note 12 a, each with its literal outcome: run of Lotse in that thread with trigger_state = received, or no run and trigger_state absent.
AG-03: Alex mentions @lotse and @texter: trigger_state = failed with multiple_agents_mentioned, no run.
AG-04: @texter in Support/Allgemein with allow_mentioned_agents = false, and with true while Texter has no write there: failed with agent_not_allowed_here both times; after createGrant of write for Texter: a guest run of Texter with agent_session_id NULL.
AG-05: Alex mentions @lotse in the main thread of Allgemein with auto_thread = true: a new thread with parent_message_id = M, operator_agent_id Lotse and title_source = auto; the run is in the new thread; M carries the trigger_state; the delivered agent.invocation names the new thread.
AG-06: Lotse runs; Alex mentions @texter (guests allowed, Texter has write) in that thread: failed with run_active_other_agent; Lotse's run unchanged, no second run.
AG-07: Lotse's Fake-Agent does not poll; Alex triggers; the Owner revokes G-1; the poll then delivers no agent.invocation; the run is failed with access_revoked, M has trigger_state = failed with access_revoked, and one audit row agent.invoke deny with stage = delivery exists.
AG-08: on one queued run of Lotse, each of empty, forged:bad_signature, forged:alg_hs256, forged:alg_none, forged:typ_access and Texter's real token for Texter's run (foreign aud, through export_token and import_token) gives 403 invocation_token_invalid with reason invalid at startRun and at renewInvocationToken, the SDK verdict invalid, and the run stays queued and unchanged; absent gives 400 validation_failed; then the Fake-Agent's default ends the run failed with invalid_invocation_token; no start reached running.
AG-09: two Fake-Agent processes of Lotse with consumers a and b both receive and accept the invocation and start in a parallel group: exactly one 200; the other gets 409 already_started or 403 invocation_token_invalid with reason superseded; one run.state_changed to running.
AG-10: Lotse runs, then crash; a new Fake-Agent process of Lotse sends heartbeats without the run; after agent.heartbeat_persist_interval and agent.presence_timeout and run_jobs runs.stall_check the run is failed with interrupted; the thread has exactly one run and no second agent.invocation; case stale: no heartbeat at all, after run.orphan_timeout the run is failed with stale.
AG-11: Lotse's runtime reports personal: verdict billing_mode_mismatch, no start request, finishRun failed with billing_mode_mismatch; case server: skip_own_checks and billing_mode_used = personal give 409 billing_mode_mismatch, the run failed with billing_mode_mismatch, one deny audit row with target, actual mode and evidence codes; case unknown: matching mode with billing_evidence.key_type = unknown gives 409 billing_mode_mismatch.
AG-12 (rows rest and gateway): a patch on /state gives 400 validation_failed with errors[].code patch_path_forbidden at pointer /ops/0 (gateway: an error frame with validation_failed); a patch with base_version: stale gives 409 base_version_conflict with current equal to getRun at that moment; the SDK rebases and the next patch succeeds; runs.version rose only for stored patches.
AG-13: Sam gets trigger on Allgemein in the setup. Case steer (heartbeat followups: steer): Sam writes while Lotse runs: one run_followups row queued, trigger_state = received, one agent.followup with audience Lotse; the Fake-Agent receives it with trigger_kind = followup, patches delivered (message followup_delivered) and absorbed; finishRun completed creates no new run. Case queue: two follow-ups, no agent.followup; after completed the earlier one by thread_seq is the triggering message of a new queued run with triggered_by its author, the later one a queued follow-up of the new run. Case stop: Alex calls cancelRun with two open follow-ups, the Fake-Agent finishes cancelled with user_cancelled: both rows discarded, both messages trigger_state = cancelled with user_cancelled, no new run. Case waiting_input (steer and queue): the Fake-Agent starts with S and calls await_input; Alex's follow-up is delivered with token A; resume with S gives 403 invocation_token_invalid with reason superseded and the run stays waiting_input; then resume with A gives 200 and running; a further resume with S gives 409 already_started.
AG-14: as the example of Design note 3, plus: after cancel_timeout, finish completed and await_input give 409 invalid_state_transition and resume gives 409 already_started.
AG-15: Lotse posts a card with two single-use buttons (restricted_to = any) for its waiting_input run; Alex and the Owner click different buttons in a parallel group: one 201, one 409 component_used; one component used, the other disabled; one interactions row; one interaction.created; one deny audit row with component_inactive.
AG-16: Lotse posts the approval card of 09 11.5 for Alex's run; Alex clicks "Always allow": 403 forbidden with reason component_restricted, nothing changes; the Owner's click on the same button of a second card gives 201.
AG-17: the Fake-Agent reports sessions ref-a and ref-b for two threads of Lotse; Alex binds with an outdated expected_current_session_id: 409 conflict with reason session_changed; Alex binds ref-b to the first thread without take_over: 409 conflict with reason session_bound_elsewhere; sessions and audit unchanged.
AG-18: an imported session row of Lotse in a thread (fixture); Alex triggers; the delivered invocation carries the session mode bind (09 9.1, as M2-12 put it into the payload); the Fake-Agent starts and reports bind_failed: session_missing with a new reference: a new current session, the bound row ends with rebind, exactly one system line, the run stays running and finishes completed.
AG-19: Alex mentions @lotse in the main thread of Support/Eskalation; Lotse runs and finishes; set_title gives 200 with title_source = auto; Alex sets a title (manual); Lotse's set_title gives 409 title_manual, title and version unchanged.
AG-20: in the setup the Owner grants Robin write and Texter write on Support/Eskalation and sets allow_mentioned_agents = true there; before Alex's trigger a thread of Eskalation holds messages of Robin (write only), of Texter as guest, of Tracker (G-6), an import message (fixture) and of the Owner; the Fake-Agent's context reports trust = unverified for Robin, Texter, Tracker and the import, authorized for the Owner, trigger for M, and every unverified text only inside a marked block of the composed prompt. Deferred: attachment, transcript.
AG-21: with agent.context_max_bytes and agent.context_message_max_bytes set low in config: long elements truncated, the sum within the limit, the oldest missing, truncated_before_seq set, the trigger complete.
AG-22: case scheduled (configuration as M2-06's AG-22 test: signing_key.cache_ttl and agent.invocation_token_ttl plus auth.clock_leeway below agent.jwks_max_age, below signing_key.invocation_rotation_interval): a repeat of trigger, receive, start, finish while the clock and run_jobs instance.key_rotation carry two complete rotations: every verdict valid, every start 200, no run failed; case emergency (Design note 12 c): one delayed delivery (redelivered: true, token of the new key), then 200, no run failed.
AG-23: Lotse's Fake-Agent on the gateway registers a command with an option autocomplete = true and mode ignore; Alex's autocompleteCommand (asynchronous step) is reported as command.autocomplete_requested by the Fake-Agent; after command.autocomplete_timeout the answer is 200 with an empty list; control case answer: the choices arrive.
AG-24: register_commands with each of stop, status, new, resume, title, close: 409 conflict with reason command_name_reserved; no commands row.
AG-25: only the deferred line voice M3-09 and the title of 09 16.2; the file has no case until M3-09 adds it.
AG-26: Lotse's Fake-Agent posts into a thread of Marketing/Kampagnen: 404 identical to an unknown thread; after the Owner grants Lotse read on Marketing: 403 forbidden with reason level_insufficient; nothing stored.
AG-27: Lotse does not poll; Alex triggers; after run.queued_max_age and run_jobs runs.queued_expiry: failed with agent_unavailable, M failed with agent_unavailable; a later poll delivers nothing for that run.
AG-28: Design note 12 d: accept and acknowledge T1; the clock passes agent.ack_timeout and agent.invocation_token_ttl plus auth.clock_leeway; a poll delivers no redelivery; start with T1 gives 403 invocation_token_invalid with reason expired; renew gives T2; start with T1 gives superseded; restart restoring the snapshot; the next poll delivers the same event again with T3, which replaces T2 in the inbox (09 4.3); start with T2 gives superseded; start with T3 gives 200 and again 200 (replay); exactly one run.state_changed to running.
AG-29: renew with a token replaced by a redelivery: 403 invocation_token_invalid with reason superseded; renew of a running run: 409 already_started; after the Owner revokes G-1: 403 agent_invocation_denied, the run failed with access_revoked, one deny audit row.
AG-30: Texter's runtime loads host, Sam triggers Texter (token tool_class = scoped): verdict tool_class_mismatch, no start, finishRun failed with tool_class_mismatch; case server: skip_own_checks with tool_class_used = host gives 409 tool_class_mismatch and the run failed with tool_class_mismatch; chat is accepted.
AG-31: Lotse starts with S, posts an approval card bound to the run and calls await_input; renew and resume with S give 403 invocation_token_invalid with reason superseded and the run stays waiting_input; Alex clicks "Allow once"; the Fake-Agent receives interaction.created with token A (interactions.state = dispatched); resume with A gives 200 and running.
AG-32: the clock passes T1's expiry; proxy drop_next renewInvocationToken; renew with T1 and retry: false fails at the transport, the conductor keeps T2; renew with T1 gives 200 with T3 and runs.renewed_from_jti is T1's jti; start with T2 gives superseded; start with T3 gives 200; one start; renew with T1 gives 409 already_started; after await_input it gives 403 invocation_token_invalid with reason superseded.

Plan row M2-31, first acceptance test ("the suite runs in make check"): TestConformance is a plain Go test of services/server, so make test (go test -race ./... per Go module) runs it inside make check; no build tag, no skip, no separate target. Second acceptance test ("every scenario was red before its implementation"): the red run of every scenario file before the conductor and the Fake-Agent supported it (Design note 14 a), TestExpectations_Bite (note 14 b) and the table of note 14 c in the report.

Contract: every human response in the suite validates against docs/api/openapi.yaml (M1-31 validator); every Fake-Agent reply that reports a problem carries code and, where named above, reason or errors[].code; no JSON null in any response or in any event_log payload the checks read.

Red team focus of the milestone (PLAN 8 "Red team": forging or reusing the invocation token; faking the reported mode; another person triggering a personal subscription) is covered end to end by AG-08, AG-11, AG-28 to AG-32 and AG-01; name them in the red team request.

Commands that must pass, from the repository root:

cd services/server && go test -race -count=1 ./test/agents/... ./test/harness/...
cd services/server && go test -race -count=1 -run 'TestConformance/AG-14' ./test/agents/ (one scenario alone, as the README describes)
cd sdk/python && uv run --locked pytest
cd tools && uv run --locked --quiet pytest ci/tests/test_components.py
make arch, make lint, make typecheck, make coverage, make docs
make check

Acceptance criteria¶

Definition of done¶

Code and tests as specified; doc comment on every public symbol (purpose, parameters, return value, errors, example where useful); comments explain why.
READMEs of services/server/test/agents (new), services/server/test/harness and sdk/python updated (scenario format, step kinds, Fake-Agent ops and control protocol, deferred parts and how a later ticket adds a part, the harness additions, running one scenario).
Rulings of Design note 16 recorded in docs/DECISIONS.md (one row each with reason); the changes to 09, 11, 02, ADR 0022 and docs/quality/gates.md of the Scope made in the same change.
services/server/CHANGELOG.md, sdk/python/CHANGELOG.md and the root CHANGELOG.md entries under 'Unreleased'.
An ADR only if you take an architecture decision beyond this brief (MADR 4 as in docs/adr/0000-template-madr-4.md, headings and text in English, next free number, make adr-index).
make check green.
An independent code review over the diff, then the red team gate (docs/quality/red-team-gate.md, verdict JA or NEIN); on NEIN fix with a test per finding and have the delta checked again. Record both verdicts.
Land the change as AGENTS.md ("How work lands on main") describes: a pull request from this branch, squash-merged only when review, red team and make check, ci-ok and the review threads are green; then git worktree remove.

Report¶

Write to {{REPORT_PATH}}:

First line exactly Verdict: DONE or Verdict: BLOCKED <reason>.
Commits (SHA and subject), changed files with line counts, test results with counts (the red run before the change and the green run after, including the red run of every scenario file and the table of Design note 14 c), the wall time of go test ./test/agents/..., the summary of make check.
Review and red team verdicts and the findings fixed.
What is not proven.
Follow-ups that could not be done in this scope, each with its reason, and spec gaps found; at least the deferred parts with their tickets, the AG-22 emergency case of Design note 12 c and the AG-08 "missing" interpretation of Design note 12 b.

Then write {{DONE_PATH}} with exit=0 sha=<sha> or exit=1 <reason>.