Skip to content

M2-20: Heartbeat, presence and desired versus actual mode

Status: ready | Depends on: M2-07 | Milestone: M2

Goal

After this ticket an agent adapter reports the state of its runtime with sendHeartbeat (POST /api/v1/agents/me/heartbeat) or with the member agent of the gateway op heartbeat, in the schema agent_heartbeat.v1 of 09 "Heartbeat and reported state": state, runtime version, actual billing mode (reported_billing_mode) with its evidence (billing_evidence, never a key value), the modes it can prove, the tool class it actually loaded, capabilities, active runs and project readiness. The server keeps the presence of every agent in memory, derives online, degraded or offline (offline once agent.presence_timeout has passed without a heartbeat), stores the heartbeat fields at most once per agent.heartbeat_persist_interval, stores and audits a change of the actual billing mode or tool class at once (action agent.heartbeat), and sends the ephemeral event agent.presence_changed {agent_id, state, reported_billing_mode} to every principal with agent.read when state or actual mode change. The snapshot (getSnapshot, ready) and the agent card (getAgent, listAgents) show this presence instead of the placeholder unknown of D-G-280. When actual and desired mode differ, the agent card carries the warnings of 10 "Expected and reported", and a gauge lets the alarm BothyBillingModeMismatch fire in the local observability stack once the difference has lasted billing.mismatch_alert_after. This is the server part of BR-05: the heartbeat's actual mode is evidence for display, warning, audit and alarm and never a permission; the binding checks stay in the adapter (I3) and at run.start (BR-06, M2-09), so a lying heartbeat cannot weaken any check.

Read first

  • docs/PLAN.md: section 3 "Way of working per ticket"; section 8 "M2 Agent platform, extensions and terminal" (Goal; Scope "10 with I2 to I4 and I6, configured and reported mode"; Out of scope "fencing across several nodes (M7)"; Exit criteria items 2 and 10; Test strategy; Red team "faking the reported mode"; Risks "fail closed on unknown"), block "agents and extensions" rows M2-03, M2-07, M2-08, M2-09, M2-20, M2-21, M2-29, M2-31, M2-33, M2-34; section 7 rows M1-04, M1-51, M1-63, M1-64, M1-66, M1-68; section 12 row M6-14; section 13 rows M7-09, M7-17, M7-20; section 18.3 row "Presence of agents".
  • Dependencies. Header: M2-07 (agent consumers, internal/wire/configkeys_agents.go). Implied through it and therefore not in the header: M1-10 (internal/platform/otel, meter provider), M1-11 (internal/platform/lifecycle: Component, StopStep, PhaseConnections), M1-13 (unit of work), M1-16 (event log and envelope), M1-18 (HTTP chain, problem details, closed request schemas), M1-19 (rate limits), M1-24 (auditing.Recorder), M1-25 (action registry), M1-28 (authorize, AgentSelf, AgentPolicySource with ReportedToolClass, filter and delivery mode), M1-44 (agent API tokens), M1-51 (agent records, Agent.presence, internal/agents/adapters/authzsource), M1-63 (snapshot with agents in state unknown), M1-64 (gateway op heartbeat that ignores member agent until this ticket, rate limiter port, wstest), M1-65 (hub, authorize.Delivery(ctx), two server instances in one test binary). Needed but outside the closure of the header (reported as plan gaps; see Boundaries): M1-66 (kernel/eventlog.EphemeralPublisher and the ephemeral path of the hub), M2-03 (ApplyProjectReadiness on the chat port), M1-68 (metric registry internal/platform/otel/metrics, signal check) and M1-04 (Prometheus rule tests and the alert-thresholds check).
  • docs/spec/09-agents.md: 2.2 "Features" (capability names of the heartbeat), 2.4 "Tool class" (warning on a different actual class), 4.1 "Overview", 4.2 "Transports", 4.7 "Adapter operations" (row Heartbeat: "last one wins"), 4.8 "Heartbeat and reported state" (all), 5.5 "Stall, interruption, orphaned runs" (M2-09 reads what this ticket stores), 15 "Reliability", 16.1 "Test doubles", 17.8 "Reported mode", 18.3 "Reported mode per run" (last paragraph), 19 (R-AG-01), 21 "Configuration values of this chapter".
  • docs/spec/10-backend-policy.md: 3 "Invariants" (I3, I5), 7 "Adapter check (I3)" (strength of the evidence sources; an undeterminable mode counts as unequal to every desired mode), 8 "Expected and reported" (both tables and the audit paragraph), 9.2 "What happens when switching" point 5, 13 "Display in the app" (row Agent card (console)), 14 "Audit trail" (row "Reported mode changed"), 15 "Tests" (BR-05, BR-06, BR-13), 17 "Configuration values of this chapter" (billing.mismatch_alert_after).
  • docs/spec/04-sync-protocol.md (core contract, read only): 2.1 "Two kinds", 2.2 "Envelope", 6.2 "Ops" (row heartbeat; error keeps the connection), 6.3 "Lifecycle" steps 4 and 5, 6.4 "Close codes" (4002, 4009), 7.2 "Flow" (ephemeral frames only after catchup.complete), 9 "Delivery guarantee", 12 "Typing and agent presence" items 2 and 3, 17 (R-SY-25), 18 "Configuration values of this chapter" (agent.presence_timeout, sync.heartbeat_interval).
  • docs/spec/03-data-model.md (core contract, read only): table agents (columns reported_tool_class, reported_runtime_state, reported_billing_mode, reported_runtime_version, reported_details, last_heartbeat_at, version; index (last_heartbeat_at); the paragraph on agent.heartbeat_persist_interval below the table), audit_events.details, 6.3 "Lock order", 9 "Configuration values of this chapter" (agent.heartbeat_persist_interval).
  • docs/spec/05-actions.md (core contract, read only): 1 "Binding rules" rules 1 to 4; 2.1 "Column "Permission"" (agent-self); 2.4 (column Audit); 2.5 "Decision" (rate limit before authorize); 3 "Rate-limit classes" (rl.agent_runtime); 4.10 rows agent.read and agent.heartbeat; 6, table "Ephemeral events", row agent.presence_changed; 7 row heartbeat; 8 R-AZ-04.
  • docs/spec/02-architecture.md: 3.2 "Roles of a process"; 4.1 "Rules"; 4.3 (row agents with port Presence; row gateway implementing eventlog.EphemeralPublisher); 4.4 (agents may use identity, authz, chat; gateway and sync may use agents); 4.5 (kernel/eventlog, kernel/auditing, kernel/clock, kernel/errs); 4.6 and 4.7 (owners of agent.heartbeat, agent.presence_changed, op heartbeat); 5 "Layers within a module"; 8.1 "Flow"; 11.6 "Ephemeral events"; 11.7 "Cost of serialization"; 15.3 "Shutdown"; 16 "Scaling path" ("Presence and typing: via NOTIFY ephemeral").
  • docs/spec/06-api.md: 2 "Basic rules" (row Unknown values), 5 "Errors" (point 3: errors[] with not_enabled; point 6), 7 "Idempotency" (sendHeartbeat exempt), 9 "Rate limits", 11 "Real-time gateway", 16 "Endpoints" (row sendHeartbeat), 17 "WebSocket ops and events" (rows heartbeat and agent.presence_changed).
  • docs/spec/07-permissions.md: 2.9 "Agents as principals", 2.11 "Tool class of an agent", 4.2 "Flow" (filter and delivery mode write no audit), 8.1 "Delivery filter" (row "Ephemeral events"), 11.2 "Required tests" (AZ-P1, AZ-P2, AZ-P5, AZ-P13).
  • docs/spec/01-product.md: feature rows "Presence of agents" and "Presence of people", wireframe A5, R-PR-02. docs/spec/20-privacy.md row "Presence". docs/spec/17-observability.md: 2 rules 3, 4, 8, 9; 6 "SLOs and alerts". docs/spec/18-release.md 9 point 5. docs/spec/19-security.md SEC-AUD-5, SEC-IN-1.
  • docs/api/openapi.yaml: operation sendHeartbeat; schemas Heartbeat, HeartbeatAck, BillingEvidence, BillingMode, ToolClass, AgentRateLimitState, ProjectReadiness, JsonObject, AgentRuntimeStatus, AgentRuntimeState, Agent, AgentPresence, Snapshot, GatewayReady, GatewayHeartbeat, GatewayHeartbeatAck, GatewayError, AgentPresenceChangedEvent, AgentPresencePayload, EventEnvelope, AuditVerifyResult (pattern of a warnings member), Problem with x-problem-codes. docs/api/asyncapi.yaml: operation receive.heartbeat, message event.agent.presence_changed.
  • ADRs 0008, 0013, 0017, 0022, 0024, 0034, 0035, 0050, 0052, 0053.
  • docs/DECISIONS.md: D-L8, D-K-A13, D-K-D29, D-V08 (field rate_limit, built by M6-14), D-G-280 (this ticket ends its presence part), D-W-100; section 10 for the format of new rows.
  • docs/security/threat-model.md: 5.10 "Agent adapters and runtime", register rows T-AI-08 and T-TK-02. docs/RISKS.md: RK-13, RK-37. docs/OPEN-QUESTIONS.md: OQ-06 (default "leave it as it is": humans have no presence), OQ-07 (default M7; "The code stays node-neutral from M1 on").
  • docs/observability/signals.md: 1 "Names and units" item 4, 2 "Cardinality", 4.3 rows bothy.agent.heartbeat.age and bothy.agent.billing_mode_mismatch, 5.3 row bothy.agent.handle, 7 "Logs". docs/observability/alerting.md: 3.2 rows BothyAgentOffline and BothyBillingModeMismatch, 4 anchors. docs/observability/dashboards.md row agent-runs.json. Files infra/observability/prometheus/rules/bothy-alerts.yaml and infra/observability/grafana/dashboards/agent-runs.json (panels "Heartbeat age", "Billing mode mismatches").
  • docs/operations/configuration.md: agent.presence_timeout, agent.heartbeat_persist_interval, billing.mismatch_alert_after, sync.heartbeat_interval, shutdown.total_timeout, http.max_json_body_bytes, ratelimit.rl.agent_runtime.per_minute, ratelimit.rl.agent_runtime.burst, cluster.mode.
  • M1 briefs whose interfaces this ticket uses: M1-11-lifecycle-health.md, M1-13-unit-of-work.md, M1-24-audit-chain.md, M1-28-authz-core.md (AgentPolicy, "agent policy facts are read fresh per call"), M1-51-agents-records.md ("presence view", billing_mode_matches), M1-63-sync-catch-up.md (note 9, bullet agents), M1-64-gateway-connection.md (note 14), M1-65-gateway-delivery.md (candidates of an event with empty scope), M1-66-gateway-focus-typing.md (EphemeralEvent, EphemeralPublisher, ephemeral rules of the hub), M1-68-core-metrics.md (registry and recording pattern), M1-04-ops-config-checks.md (rule tests, alert-thresholds). M2 briefs for the borders: M2-03-project-binding.md (note 7, ApplyProjectReadiness), M2-07-agent-consumers.md (config key file, rate limit of a gateway op), M2-09-run-lifecycle.md (note 10, reads last_heartbeat_at and reported_details.active_runs).
  • Merged code on branch main (read with git show main:<path>, no checkout): services/server/architecture/modules.yaml (module agents: action agent.heartbeat, ephemeral event agent.presence_changed; module gateway: op heartbeat), services/server/internal/agents/README.md ("Notes for the build": heartbeat state in memory), services/server/internal/wire/configchapters.go, configkeys_messaging.go, configchecks.go.
  • Code: internal/agents (M1-51, M2-07), internal/gateway (M1-64 to M1-66, M2-07), internal/sync (snapshot of M1-63), internal/chat (port of M2-03), internal/authz, internal/kernel/authorize, internal/kernel/auditing, internal/kernel/eventlog, internal/kernel/clock, internal/kernel/ids, internal/kernel/errs, internal/platform/lifecycle, internal/platform/otel, internal/platform/pg/pgtest, internal/identity/adapters/identitytest, internal/gateway/adapters/wstest, internal/wire, services/server/api/operations.yaml.

Fixed requirements that apply

Owner's fixed build rules (apply verbatim):

  1. No MVP. New from scratch and directly in the final state; stages are only the order.
  2. Strictly modular: clear module and service boundaries, dependencies only through defined interfaces, no cross access.
  3. Documentation in code: every public module, class, function and interface with a docstring/doc comment (purpose, parameters, return value, errors, example where useful); comments explain the why, not the what. API documentation is generated from code (OpenAPI from the routes, dartdoc, Python API docs in MkDocs).
  4. Documentation in the repo: README per module/service, ADR for every architecture decision, CHANGELOG per release, keep the documentation site current. A change without matching documentation is not done.
  5. Quality gates (SonarQube, linters, type checking, security scanners, test coverage) block the merge.

Rules for every brief:

  1. Test first: write the failing test, run it and keep the red output for the report, then make it green, then clean up.
  2. No en dash and no em dash in any file you write; the text-rules pre-commit hook rejects them. Use a hyphen or rephrase.
  3. White label: examples and test data use example.com and the people Alex, Sam and Robin; no company, host or personal names, no paths under a home directory.
  4. Limits, windows and thresholds only through named configuration keys listed in docs/operations/configuration.md; code comments and docs name the key, never the number.
  5. Specification first: a change to behaviour, API, event, right or configuration updates docs/spec/ or docs/api/ in the same change. The core contract (docs/spec/00-glossary.md, 03-data-model.md, 04-sync-protocol.md, 05-actions.md) is not edited by this ticket.

Product requirements implemented here:

  1. The adapter sends the heartbeat over REST (sendHeartbeat) or as member agent of the gateway op heartbeat, with the same effect; action agent.heartbeat (agent-self), rate-limit class rl.agent_runtime, exempt from Idempotency-Key, the last heartbeat wins (05 4.10 and 7; 06 7 and 16; 09 4.7 and 4.8; openapi sendHeartbeat, GatewayHeartbeat).
  2. Content after agent_heartbeat.v1: state, versions, actual billing mode, provable modes, evidence, actual tool class, capabilities, active runs, project readiness; never secrets, contents or paths, never a key value or prefix (09 4.8; 03 agents.reported_details; 10 7; I5).
  3. Presence is derived by the server: online or degraded from the heartbeat, offline after agent.presence_timeout without heartbeat, unknown before the first one; humans have no presence (04 12 item 2; 09 4.8 row state; 01 R-PR-02; 20 row "Presence"; OQ-06).
  4. On a change the server sends the ephemeral agent.presence_changed {agent_id, state, reported_billing_mode} to every principal with agent.read; no seq, at most once, never before catchup.complete (04 2.1, 7.2, 9, 12 item 2; 05 6 "Ephemeral events"; 02 11.6; openapi AgentPresenceChangedEvent).
  5. Heartbeat fields are stored at most once per agent.heartbeat_persist_interval; a change of reported_billing_mode or reported_tool_class is stored at once and audited with action agent.heartbeat, decision allow, old and new values and the evidence method (03 agents; 04 12 item 2; 05 4.10 column Audit; 10 8 and 14).
  6. No rights version and no access.changed (05 4.10: Permissions no); authorize counts the higher of tool_class and reported_tool_class (03 agents.reported_tool_class; 07 2.11; 09 2.4).
  7. The snapshot holds the presence of the agents the principal may read, and ready carries the current state (04 6.3 step 4 and 12 item 2; openapi Snapshot.agents, AgentPresence).
  8. The server compares actual and desired mode at each heartbeat and shows the warnings of 10 "Expected and reported" on the agent card; an undeterminable actual mode counts as unequal to every desired mode (09 4.8; 10 7, 8 and 13; 09 2.4 for the tool class).
  9. A difference that lasts longer than billing.mismatch_alert_after raises an alarm (10 8 row 1; 17 6; alerting.md BothyBillingModeMismatch).
  10. Telemetry: only catalogue names, closed attribute sets, no person attributes, never blocking (17 2 rules 3, 4, 8, 9; signals.md 2).
  11. No JSON null in any body, frame or stored detail (ADR 0053); every body and frame conforms to its schema (06 15).

Scope

All paths under services/server/ unless noted. No migration: M1-51 created the table agents with every column this ticket writes. No new top-level Makefile target.

Create:

  • internal/agents/presence.go (port, doc comment on every symbol):
    • type Presence interface { Heartbeat(ctx context.Context, body json.RawMessage) (PresenceStatus, error); Statuses(ctx context.Context, agentIDs []ids.ID) ([]PresenceStatus, error) }. Heartbeat takes the JSON object of schema Heartbeat (the REST body or the member agent of a gateway frame), so both transports share one decoder (Design note 1); the actor comes from the context.
    • type PresenceStatus struct { AgentID ids.ID; State string; ReportedBillingMode string; RuntimeVersion string; LastHeartbeatAt time.Time; BillingModesSupported []string; ReportedToolClass string; BillingModeMatches bool; BillingEvidence *BillingEvidence; Warnings []string } (empty string and zero time mean "not reported"), type BillingEvidence struct { Provider, KeyType, Method string }, and the constants of the states and warning codes.
  • internal/agents/domain/heartbeat.go: DecodeHeartbeat(raw []byte) (Report, []FieldError) with the rules of Design note 3; ReportedChange(stored, next Reported) (AuditDetails, bool) (Design note 4); the reported_details form of Design note 6.
  • internal/agents/domain/presence.go: StateAt(lastHeartbeat time.Time, reported string, now time.Time, timeout time.Duration) string; StartRule(stored StoredPresence, start time.Time, timeout, persistInterval time.Duration) (offlineNow bool, deadline time.Time) (Design note 7); Assess(target Target, report Report, state string) Assessment with the mismatch kinds and warnings of Design note 8.
  • internal/agents/app/presence.go: the process-wide tracker (entries, per-agent lock, deadline timers on kernel/clock, last published pair, mismatch start times), implementing agents.Presence; internal/agents/app/heartbeat.go: the use case of Design note 2; internal/agents/app/presence_flush.go: start load, flush loop and shutdown flush (Design notes 6 and 7); internal/agents/app/metrics.go: outbound metrics port with a no-op default (pattern of M1-68).
  • internal/agents/adapters/postgres/queries/presence.sql (sqlc): GetPresenceRow (desired billing_mode, tool_class and the stored reported columns of one agent), LockPresenceRow (FOR UPDATE), WriteReportedChange, PersistHeartbeat (monotone, Design note 6), ListLivePresence (agents stored as online or degraded), GetTargets (desired values of given agents); the repository methods.
  • internal/agents/adapters/httpapi/heartbeat.go: the strict-server method SendHeartbeat.
  • The two gauges of Design note 9: as Specs in internal/platform/otel/metrics if M1-68 is merged, otherwise in internal/agents/adapters/otelmetrics/presence.go.
  • infra/observability/prometheus/tests/bothy-alerts.test.yaml (repository root): the cases of the Tests section. M1-04 creates this file and the alert-thresholds check; if either is missing at build time, create the file in the promtool test rules format M1-04 prescribes and name the missing check in the report.

Change:

  • internal/agents/app, use cases of M1-51: after a successful commit of SetBillingMode, SetBillingModeAll, SetToolClass, Disable and Enable, notify the tracker (Design note 13); getAgent and listAgents build Agent.presence from Presence.Statuses (Design note 12).
  • internal/gateway/app/heartbeat.go (M1-64): the agent part of the op heartbeat (Design note 10).
  • internal/sync (snapshot of M1-63, note 9 bullet agents): fill AgentPresence from agents.Presence.Statuses; remove the unknown placeholder and its comment.
  • internal/wire: one tracker per process in the roles api and all as a lifecycle.Component (start load, flush loop) with the stop step of Design note 6; bind Presence into the gateway, the sync snapshot and the agents HTTP handlers; bind kernel/eventlog.EphemeralPublisher (M1-66), the chat readiness port (M2-03), the identity Directory, the clock and the metrics adapter into the tracker; the configuration keys and the cross-check of Design note 16 (configkeys_agents.go of M2-07, configchapters.go, configchecks.go).
  • architecture/modules.yaml: only if the fallback of Design note 9 adds internal/agents/adapters/otelmetrics, the vendor entry it needs; make arch stays green.
  • api/operations.yaml: move sendHeartbeat from later to enabled (sorted); rerun the route generator (make codegen-check shows no diff afterwards).
  • docs/api/openapi.yaml (spec first): on sendHeartbeat a description of Design notes 2 to 5, 8 and 11 and a response example with presence.state: online and interval_ms equal to the default of sync.heartbeat_interval in milliseconds; on Heartbeat a pattern on runtime_version (version form of Design note 3) and descriptions of details (closed key set and shapes of Design note 3), project_readiness (Design note 11) and rate_limit (not accepted until M6-14); on BillingEvidence a pattern on provider and method (code form of Design note 3); on AgentRuntimeStatus the new members billing_evidence ($ref: BillingEvidence) and warnings (array of strings; description with the known values of Design note 8, the pattern of AuditVerifyResult.warnings); on HeartbeatAck.interval_ms the description "sync.heartbeat_interval"; on GatewayHeartbeat.agent a description of Design note 10. Then make api-lint, make api-tables, make api-checks.
  • docs/api/asyncapi.yaml: one sentence in the summary of receive.heartbeat: errors of the agent part answer error and keep the connection.
  • docs/spec/09-agents.md (English, like the chapter): 4.8 aligned with the openapi member names (top-level members and keys of details, project_readiness instead of projects with workspace_ref), the forms of Design note 3, full-state semantics, presence derivation, start rule, the comparison and warnings of Design note 8; 2.2 names details.capabilities as the report.
  • docs/spec/10-backend-policy.md 8: the warnings as codes of AgentRuntimeStatus.warnings, the gauge and the alarm of Design note 9, the unknown mode as a difference.
  • docs/spec/02-architecture.md 15.3 role api step 3: one sentence "Store the presence state of the agents, without an event" (Design note 6).
  • docs/spec/18-release.md 9 point 5: the adapter reports its version in details.adapter.version (stored in agents.reported_details); reported_runtime_version is the runtime's version (Design note 3).
  • docs/observability/signals.md 4.3: new row bothy.agent.billing_mode.mismatch_age (Design note 9). docs/observability/alerting.md 3.2: row BothyBillingModeMismatch with the new condition and the configuration value `billing.mismatch_alert_after`; 4 anchor bothybillingmodemismatch: one sentence on the heartbeat cause. infra/observability/prometheus/rules/bothy-alerts.yaml: the extended rule. infra/observability/grafana/dashboards/agent-runs.json: a panel "Reported billing mode differs" in the row "Adapters and policy"; docs/observability/dashboards.md row agent-runs.json: "mode mismatches (heartbeat and I3)".
  • docs/operations/configuration.md: make config-reference, then make config-doc-check.
  • READMEs of agents, gateway and sync (English, like the existing files); services/server/CHANGELOG.md under "Unreleased"; docs/DECISIONS.md section 10 rows D-M2-20-1 and following (Design note 17) and, in row D-G-280, the reference to the row that ends its presence part.

Out of scope

  • The comparison of active_runs with the server's runs (interrupted) and the missing heartbeat (stale) in runs.stall_check, the read Agents.HeartbeatStates, the I3 check at run.start with 409 billing_mode_mismatch, run reasons and the counter bothy.agent.billing_mode_mismatch (M2-09). This ticket stores last_heartbeat_at and reported_details.active_runs in the shape M2-09 reads.
  • The heartbeat field rate_limit, its storage and the backend view (M6-14, D-V08).
  • Presence across several api nodes over NOTIFY ephemeral with cluster.mode = multi (M7-17). This ticket keeps the code node-neutral (OQ-07) with monotone writes.
  • Status page components from presence (M7-09, M7-21) and alarms into a channel of the instance (M7-20).
  • Reading effective capabilities (R-AG-01) for agent.invocation, follow-ups, components, titles, voice and files: the first consumer adds the accessor (M2-08, M2-11, M2-15, M2-18, M3-09); this ticket validates and stores capabilities only.
  • The adapter side of BR-05 (M2-33 Hermes Agent adapter, M2-34 and M2-35 Claude Code runner), the heartbeat building blocks of the SDKs (M2-29, M2-30), conformance scenarios (M2-31).
  • Switching the desired mode and the guided flow (M2-21); project bindings and their readiness rules (M2-03); readiness detection in the runner (M2-34).
  • Presence and warnings in the clients and the console (M4-19, M6), including the "Presence history" of wireframe A5, which has no data model in the specification (spec gap).
  • Changes to the core contract.

Boundaries

  • Work in a dedicated git worktree on the branch feat/M2-20-heartbeat-presence; one writer per worktree.
  • Conventional Commits with the module as scope (for example feat(agents): ..., feat(gateway): ..., feat(sync): ..., docs(api): ..., feat(observability): ...) and a DCO sign-off on every commit (git commit -s); squash into meaningful commits before the merge (docs/quality/push-discipline.md).
  • Never push to main, never tag and never create remote repositories; the only remote steps are the branch push and the pull request of AGENTS.md.
  • Do not modify files outside the scope above, except the CHANGELOG, README and docs/DECISIONS.md entries named in this brief.
  • Follow the host rules supplied by the launcher (ports, machine rules).
  • Tests bind only 127.0.0.1 and port 0; PostgreSQL runs only as a throwaway container on a random port through pgtest (P-09, P-10). Without a Docker-compatible runtime the integration tests fail; they never skip.
  • Stop only processes you started, by PID. Temporary directories only via mktemp -d, removed at the end.
  • Never read or print secrets. No network calls except package downloads for pinned tools and Go modules.
  • No agent runtime, adapter, SDK, coding agent or model runs in any test and no real third-party service is called: the agent side is the wstest client of M1-64 or plain HTTP with the agent's API token and DPoP proofs from identitytest. Tests read metrics with an in-process SDK reader and alert rules only through the pinned promtool of M1-04; no collector, Prometheus or Alertmanager is started.
  • Time-dependent tests set short values for agent.presence_timeout, agent.heartbeat_persist_interval and billing.mismatch_alert_after through the configuration loader or use the kernel/clock test clock; no test depends on a default value.
  • Never lower a threshold, disable a gate or delete a test.
  • If the core contract lacks something this ticket needs, stop with Verdict: BLOCKED <reason> instead of building a deviation (P-11).
  • If the base branch lacks kernel/eventlog.EphemeralPublisher with the ephemeral path of the hub (M1-66) or ApplyProjectReadiness on the chat port (M2-03), stop with Verdict: BLOCKED <missing ticket>: both are needed and lie outside the closure of the header. The metric registry (M1-68) and the rule tests (M1-04) have the fallbacks named in this brief.
  • This brief was written before milestone M1 closed (D-W-100). Where the merged code of an M1 ticket names a package, type, function, file or make target differently from this brief, follow the code and list each difference in the report; where it behaves differently from the specification, stop with Verdict: BLOCKED <reason>.

Design notes

  1. Placement (02 4.4, 4.6, 4.7). sendHeartbeat, the action agent.heartbeat, the table agents and the event agent.presence_changed belong to agents; the op heartbeat belongs to gateway, which may use agents. One use case in agents/app behind the port Presence serves both transports: the REST handler in agents/adapters/httpapi passes the request body (re-encode the generated request object or read the raw body, whichever the merged strict server allows) and the gateway passes the raw member agent; domain.DecodeHeartbeat is the only mapping from the wire form. The presence state lives in a process-wide tracker in agents/app, started only in the roles api and all, because only they receive heartbeats (02 3.2). agents publishes through kernel/eventlog.EphemeralPublisher (02 4.5; 02 11.6: "directly to the hub") and reaches project readiness only through the chat port. Ruling for docs/DECISIONS.md.
  2. Use case Heartbeat, in this order (the rate limit rl.agent_runtime runs before, in the chain or the gateway, 05 2.5): a. authorize(actor, agent.heartbeat, {kind: agent, id: actor, owner: actor}). People and apps get forbidden with reason subject_kind_not_allowed from AgentSelf (M1-28); the denial is audited by authz (R-AZ-04). The body carries no agent id, so an agent can only report for itself. b. DecodeHeartbeat and validation (note 3); every violation returns one errs validation error with all field errors. c. Take the tracker entry's lock; read GetPresenceRow (desired billing_mode and tool_class, stored reported_billing_mode, reported_tool_class, reported_runtime_state, last_heartbeat_at) on every heartbeat, so a desired value changed in another process counts at once; create the entry if missing (handle from the identity Directory). d. Project readiness (note 11). e. Change of the actual mode or class (note 4). f. Update the entry: report, last heartbeat = now, state from the report, dirty unless step e wrote it; reassess (note 8); re-arm the deadline (note 5). g. After any commit of step e: publish agent.presence_changed if the pair changed (note 5). h. Answer 200 HeartbeatAck {presence: AgentRuntimeStatus of the entry, interval_ms: sync.heartbeat_interval}.
  3. Decoding and forms of agent_heartbeat.v1 (agents/domain, 09 4.8; the openapi schema stays the contract of the top-level members):
    • Top-level members as Heartbeat: state (online or degraded), runtime_version, reported_billing_mode, billing_modes_supported, billing_evidence, reported_tool_class, rate_limit, details, project_readiness. An unknown member is a validation error (06 2 row Unknown values).
    • details is a closed object: capabilities (object; keys of 09 2.2: followups with steer or queue, components, voice_input, title, files_out as booleans), active_runs (run ids, UUIDs after R-DM-01 checked with kernel/ids), degraded_reasons (codes), adapter (name required, version optional), runtime (name required), protocol_version (integer, at least 1). Any other key, also inside capabilities, adapter and runtime, is an unknown member. runtime_version is the runtime's version and fills reported_runtime_version; details.adapter.version is the adapter's version (18 9 point 5 is aligned to this). Ruling for docs/DECISIONS.md.
    • Forms: names (details.adapter.name, details.runtime.name) ^[a-z0-9][a-z0-9._-]*$; versions (runtime_version, details.adapter.version) ^[0-9A-Za-z][0-9A-Za-z.+_-]*$; codes (billing_evidence.provider, billing_evidence.method, each entry of degraded_reasons) ^[a-z0-9][a-z0-9_.:-]*$, the code form of SEC-AUD-5, so method enters the audit details unchanged. No duplicates in billing_modes_supported, active_runs, degraded_reasons and among the channel_id values of project_readiness (pointer /project_readiness, as M2-03 expects). Lengths are bounded by http.max_json_body_bytes. Reason: 09 4.8 forbids secrets, contents and paths, and these forms make path separators, whitespace, control and bidi characters and most key formats unrepresentable. Ruling for docs/DECISIONS.md.
    • rate_limit present: field error not_enabled at /rate_limit (06 5 point 3) until M6-14 builds the field.
    • Field error codes and pointers follow what M1-18 uses for the same kind of violation; REST answers 400 validation_failed with errors[]. The domain checks every rule of this note itself, also those the chain or the gateway decoder already enforce from the openapi patterns (SEC-IN-1), so both transports reject the same bodies.
  4. Full state and change of the actual mode or class (09 4.7 "last one wins"; 10 8; 03 agents):
    • Every heartbeat replaces the whole reported state; an absent optional member means "not reported": reported_billing_mode NULL (D-L8), reported_tool_class NULL, key absent in reported_details. Exception: project_readiness (note 11). Ruling for docs/DECISIONS.md.
    • A heartbeat whose reported_billing_mode or reported_tool_class differs from the stored value (also from or to "not reported", also the first report ever) is a change: one unit of work runs LockPresenceRow, compares again, and if still different runs WriteReportedChange (both columns, reported_runtime_state, reported_runtime_version, reported_details, last_heartbeat_at = now()) and records through auditing.Recorder in the same transaction: action agent.heartbeat, decision allow, actor the agent, resource {kind: agent, id}, details only billing_mode_old, billing_mode_new, tool_class_old, tool_class_new (for the changed column, each only when the value exists), key_type and method from billing_evidence when present. One entry per heartbeat, also when both columns change. Equal after the lock: nothing written, no audit (another process stored it). Ruling for docs/DECISIONS.md.
    • Only this path writes reported_billing_mode and reported_tool_class; the flush of note 6 never does, so neither column changes without its audit entry.
    • Nothing else happens: no agents.version change (it is the ETag of the record, and owners' If-Match writes must not fail on heartbeats), no agent.updated, no permission_version, no access.changed, no event_log row (05 4.10; 02 11.7). authorize reads agent policy facts fresh on every call (M1-28), so a new reported_tool_class counts for SR2 from the commit on and can only raise the effective class.
  5. Presence state, deadline and event (04 12 item 2):
    • States: unknown (no heartbeat ever, the column default), online or degraded (the state of the latest heartbeat), offline once now >= last heartbeat + agent.presence_timeout without a newer heartbeat. Ruling for docs/DECISIONS.md: the boundary is inclusive.
    • Each entry has one timer of kernel/clock at its deadline; every heartbeat re-arms it. When it fires, under the entry lock and only if no newer heartbeat arrived: state offline, entry dirty, mismatch start times cleared (note 8), then publish.
    • agent.presence_changed is published exactly when state or reported_billing_mode differs from the last published pair of the entry (initially the stored values). Envelope through EphemeralPublisher.PublishEphemeral: type agent.presence_changed, schema_version from the event register, actor {id: agent, kind: agent} for a heartbeat and none for the timer, empty scope, resource {kind: agent, id} without version, no audience, payload AgentPresencePayload {agent_id, state, reported_billing_mode}; the last reported mode stays in the payload while offline (03: "last reported mode") and is omitted when not reported. The hub admits candidates with receive action agent.read in authorize.Delivery(ctx) (M1-65 candidates of an event with empty scope: all connections; M1-66 ephemeral rules: never before catchup.complete, dropped at a full queue, never 4014). A publish error is logged with the agent id and never fails the heartbeat. Ruling for docs/DECISIONS.md: the trigger pair and the mode while offline.
  6. Persistence (03 agents: at most once per agent.heartbeat_persist_interval):
    • A flush loop of the tracker runs every agent.heartbeat_persist_interval and writes each dirty entry, ascending agent id, in one unit of work per run, with PersistHeartbeat: UPDATE agents SET reported_runtime_state = $state, reported_runtime_version = $version, reported_details = $details, last_heartbeat_at = $hb WHERE principal_id = $id AND (last_heartbeat_at IS NULL OR last_heartbeat_at <= $hb). No row updated means a fresher heartbeat is stored already; the entry becomes clean either way. The offline write is the same statement with the entry's last heartbeat time, so it never overwrites a newer heartbeat stored by another process. Ruling for docs/DECISIONS.md.
    • reported_details holds exactly capabilities, billing_modes_supported, billing_evidence, active_runs (array of run id strings), degraded_reasons, adapter, runtime, protocol_version, absent members omitted, never null; rate_limit stays out until M6-14; project readiness is stored only in project_bindings.ready (M2-03).
    • These statements take no counter, no NOTIFY and no audit (02 11.7).
    • Shutdown: a stop step agent_presence in lifecycle.PhaseConnections, registered after the gateway's step so no gateway heartbeat arrives later, stops the timers without publishing and flushes every dirty entry within the rest of shutdown.total_timeout; a server restart is not an agent going offline. Ruling for docs/DECISIONS.md; add the sentence to 02 15.3.
  7. Start (lifecycle.Component of the tracker, roles api and all): ListLivePresence returns agents stored as online or degraded; keep those whose principal status is active (identity Directory, which also gives the handle). If last_heartbeat_at is missing or older than agent.presence_timeout plus agent.heartbeat_persist_interval, the agent is offline at once (the stored value lags the real last heartbeat by at most the persist interval, so no heartbeat came within agent.presence_timeout); otherwise it is tracked with the deadline start plus agent.presence_timeout, which gives adapters time to reconnect after 4013. Agents not loaded get an entry with their first heartbeat (note 2c). Ruling for docs/DECISIONS.md.
  8. Desired versus actual (10 7 and 8; 09 2.4 and 4.8), computed by domain.Assess from the desired values of note 2c or note 13, the last report and the state:
    • billing_mode_matches stays as M1-51 builds it from the reported mode.
    • Difference while the state is online or degraded: (a) reported_billing_mode present and unequal to billing_mode; (b) reported_billing_mode absent, billing_modes_supported reported and billing_mode not in it; (c) billing_evidence.key_type = unknown (10 7: an undeterminable mode counts as unequal to every desired mode). The tracker keeps the time a difference of any kind began and the time (a) began, and clears them when the condition ends or the state is offline or unknown.
    • AgentRuntimeStatus.warnings, only while online or degraded: billing_mode_mismatch when (a) has held longer than billing.mismatch_alert_after (10 8 row 1 names the duration); billing_mode_unsupported when (b) holds (row 2); billing_evidence_unknown when (c) holds (row 3); tool_class_mismatch when reported_tool_class is present and unequal to tool_class (09 2.4). A warning disappears with the heartbeat that ends its cause (10 9.2 point 5). Ruling for docs/DECISIONS.md: codes, the delay only for (a), no warnings while not present.
    • The server takes no action on runs: the binding checks are in the adapter (I3) and in run.start (BR-06, M2-09).
  9. Metrics and alarm (signals.md; alerting.md; 17 rule 4):
    • bothy.agent.heartbeat.age (catalogue row, Gauge, s, bothy.agent.handle): for every active tracked agent with a known last heartbeat, now minus that heartbeat.
    • New catalogue row bothy.agent.billing_mode.mismatch_age (Gauge, s, attribute bothy.agent.handle, Source "api: seconds since the start of a mismatch between configured and reported mode according to the heartbeat (10, 8); 0 without a mismatch", Prometheus bothy_agent_billing_mode_mismatch_age_seconds): for every active tracked agent, now minus the start of a difference of any kind of note 8, else 0. Ruling for docs/DECISIONS.md (the catalogue has no signal for the heartbeat difference; spec gap).
    • Both are observable gauges that read only tracker memory, in the roles api and all; the values of bothy.agent.handle come from the tracked agents, a set the owner limits (signals.md 2 item 1); disabled agents have no series. Declare them as Specs in internal/platform/otel/metrics with their attribute set if M1-68 is merged; otherwise in internal/agents/adapters/otelmetrics over the meter of M1-10 with the vendor entry in architecture/modules.yaml, and name in the report that the registry takes them over, as M2-01 does.
    • Rule BothyBillingModeMismatch in bothy-alerts.yaml: keep the existing counter expression and add or max by (bothy_agent_handle) (bothy_agent_billing_mode_mismatch_age_seconds{job="bothy/bothy-server"}) > followed by the default of billing.mismatch_alert_after in seconds, as the last comparison of the expr, because the alert-thresholds check of M1-04 reads the final comparison. Severity page stays. alerting.md row: condition "reported mode differs from the configured mode, according to the heartbeat for longer than the threshold or at a run (I3)", configuration value `billing.mismatch_alert_after`.
    • Dashboard panel: max by (bothy_agent_handle) (bothy_agent_billing_mode_mismatch_age_seconds{job="bothy/bothy-server",deployment_environment_name=~"$env",bothy_agent_handle=~"$agent"}).
    • Recording never blocks and never changes a heartbeat's result.
  10. Gateway (internal/gateway/app/heartbeat.go; 04 6.2; 05 7 row heartbeat):
    • The op keeps action gateway.connect and the liveness rules of M1-64 first (heartbeat_ack, reset of the 4009 deadline). Then, if member agent is present: rate limit rl.agent_runtime keyed by the principal through the gateway's rate limiter port (as M2-07 does for ack), then agents.Presence.Heartbeat with the raw member in the session's actor context.
    • Errors of the agent part answer error {op: "heartbeat", code} and keep the connection: forbidden (people and apps), validation_failed, rate_limited, internal_error. Violations of the frame schema found by the decoder of M1-64 still close with 4002.
    • A heartbeat of an agent without member agent keeps the connection alive and changes no presence: presence needs the state report, and 04 12 item 2 "implicitly through the gateway heartbeat" refers to this member. Ruling for docs/DECISIONS.md.
  11. Project readiness (openapi Heartbeat.project_readiness; M2-03 note 7). Only listed channels are applied; when the member is absent or the list equals the last list this process applied for the agent, nothing is called; otherwise ApplyProjectReadiness(ctx, agentID, reports) of the chat port runs in its own unit of work before the transaction of note 4, so no transaction holds rows of two modules (03 6.3). Duplicates are rejected by note 3 already; any other error of the port fails the heartbeat with 500 and changes no presence. M2-03 applies only reports of the effective operator and ignores others without error; this ticket adds no rule. Ruling for docs/DECISIONS.md: change-only call, separate transaction.
  12. Snapshot and agent card. Presence.Statuses answers tracked agents from memory and other agents from their row (unknown or offline as stored, with the stored mode, version and details); it does not authorize: callers pass only agents the actor may read (sync in filter mode agent.read, M1-63; getAgent and listAgents after their authorize). Snapshot.agents gets AgentPresence {agent_id, state, reported_billing_mode}; getAgent, listAgents and HeartbeatAck.presence get AgentRuntimeStatus with billing_evidence and warnings. Only principals of kind agent ever have presence (R-PR-02). D-G-280 ends for presence; active_runs stays with M2-05.
  13. Desired values changed. After the commit of SetBillingMode, SetBillingModeAll, SetToolClass, Disable and Enable (M1-51, same module), the use case calls the tracker with the affected agent ids; the tracker reads GetTargets for tracked ones and reassesses note 8. Disable marks the entry inactive (no metric series; the deadline keeps running, so the agent becomes offline after agent.presence_timeout, its tokens being revoked); Enable makes it active again. No event: M1-51 already writes agent.updated. Another process learns the new values with the next heartbeat (note 2c).
  14. Errors (06 5; codes only from x-problem-codes):
Situation REST sendHeartbeat Gateway, member agent
caller is not an agent 403 forbidden, reason: subject_kind_not_allowed error with forbidden; liveness counted
body or frame breaks its schema in docs/api/openapi.yaml (unknown top-level member, enum, pattern; SEC-IN-1) 400 validation_failed (chain of M1-18) what the decoder of M1-64 does for a schema violation (close 4002)
rule of note 3 that the schema does not express (details keys and forms, duplicates, protocol_version) 400 validation_failed with pointers error with validation_failed
rate_limit present 400 validation_failed, errors[].code not_enabled, pointer /rate_limit error with validation_failed
class rl.agent_runtime exhausted 429 rate_limited with Retry-After error with rate_limited
database or readiness port failure 500 internal_error, presence unchanged error with internal_error
  1. Concurrency. One lock per tracker entry serializes heartbeats, the timer and target updates of one agent in a process; the flush loop takes each entry's lock only to copy its state. Across processes the row lock of note 4 lets exactly one process audit a given change, and the monotone write of note 6 keeps the newest heartbeat. A transaction of this ticket locks at most one agents row and no row of another module (03 6.3).
  2. Configuration keys used: agent.presence_timeout, agent.heartbeat_persist_interval, billing.mismatch_alert_after, sync.heartbeat_interval, shutdown.total_timeout, http.max_json_body_bytes, ratelimit.rl.agent_runtime.per_minute, ratelimit.rl.agent_runtime.burst. Register the first three in internal/wire/configkeys_agents.go if the base lacks them (kind duration, default from their chapter tables, chapters 04, 03 and 10; add the constants chapter04 and chapter10 to configchapters.go if missing). Cross-check presence_timeout_above_heartbeat_interval in configchecks.go: agent.presence_timeout must be greater than sync.heartbeat_interval, otherwise the start fails with a message naming both keys, because an adapter that follows interval_ms would otherwise flap to offline. HeartbeatAck.interval_ms is sync.heartbeat_interval, the interval the gateway announces in hello. Ruling for docs/DECISIONS.md.
  3. Rulings for docs/DECISIONS.md section 10 (one row each with date, decision, reason and place): D-M2-20-1 one decoder and use case for both transports, tracker per process (note 1); D-M2-20-2 closed key set and forms of agent_heartbeat.v1, runtime_version versus details.adapter.version (note 3); D-M2-20-3 full-state semantics, the first report and changes from or to "not reported" are audited changes, one entry with fixed detail keys, only the change path writes the audited columns (note 4); D-M2-20-4 inclusive offline boundary, event on a change of state or reported mode, mode kept while offline (note 5); D-M2-20-5 monotone flush and shutdown flush without event (note 6); D-M2-20-6 start rule (note 7); D-M2-20-7 difference kinds, warning codes, delay only for (a), nothing while not present (note 8); D-M2-20-8 gauge bothy.agent.billing_mode.mismatch_age and the extended BothyBillingModeMismatch (note 9); D-M2-20-9 gateway order and no presence without member agent (note 10); D-M2-20-10 readiness change-only in its own transaction (note 11); D-M2-20-11 interval_ms and the cross-check (note 16); D-M2-20-12 the members billing_evidence and warnings of AgentRuntimeStatus. The offline rule, the persist interval, the immediate audited change, the receive action and the payload are in 03, 04 and 05 and need no ruling.

Tests (write them first)

Unit (go test, test clock of kernel/clock, fakes for the ports):

  • internal/agents/domain:
    • TestDecodeHeartbeat: table with the minimal body {state: online}, a full body, state: offline (rejected), an unknown top-level member, an unknown key in details, capabilities and adapter, followups: always (rejected), a non-UUID and a duplicate in active_runs, protocol_version: 0, duplicates in billing_modes_supported, degraded_reasons and project_readiness (pointer /project_readiness), rate_limit (code not_enabled, pointer /rate_limit); each rejection names its pointer.
    • TestHeartbeatForms: names, versions and codes with /, \, whitespace, uppercase letters in codes, =, a control and a bidi character are rejected at their pointers; resolved_credentials, api_key, 2.3.1, 1.0.0+build.7 are accepted.
    • TestReportedChange: from not reported to organization, organization to personal, personal to not reported, class chat to host, both at once (one detail set), equal values (no change); detail keys exactly as note 4, absent values omitted, never null.
    • TestStateAtBoundary: just before last + agent.presence_timeout the state is the reported one, at and after it offline; no heartbeat gives unknown.
    • TestStartRule: stored online with last_heartbeat_at older than agent.presence_timeout plus agent.heartbeat_persist_interval is offline at once; younger gets the deadline start plus agent.presence_timeout; missing last_heartbeat_at is offline at once.
    • TestAssess: one case per difference kind (a), (b), (c) and per warning code; billing_mode_mismatch absent exactly at billing.mismatch_alert_after and present after it; no warnings and no difference while offline or unknown; tool_class_mismatch in both directions.
    • FuzzDecodeHeartbeat: arbitrary bytes never panic, and an accepted report never contains /, \ or whitespace in a name, version or code. Its seed corpus runs in every go test; run it once locally with go test -run=^$ -fuzz=FuzzDecodeHeartbeat -fuzztime=<duration of your choice> and report duration and result; commit any crashing input as a seed after the fix.
  • internal/agents/app (fake repository, authorizer, publisher, readiness port, recorder):
    • TestHeartbeat_PublishesOnlyOnStateOrModeChange: same state and mode twice publishes once; degraded publishes; a mode change with the same state publishes; a change of degraded_reasons only does not.
    • TestHeartbeat_OfflineTimer: without a heartbeat the timer sets offline and publishes without actor; a heartbeat just before the deadline re-arms it.
    • TestHeartbeat_PersistsAtMostOncePerInterval: many heartbeats inside one agent.heartbeat_persist_interval cause one PersistHeartbeat per agent; a mode change causes one WriteReportedChange at once and one recorder call.
    • TestHeartbeat_ChangeEqualAfterLockWritesNothing: the locked row already carries the new values: no write, no audit.
    • TestHeartbeat_ReadinessOnlyOnChange: the same list twice calls the port once; a changed list calls it again; an absent member never.
    • TestHeartbeat_ShutdownFlushPublishesNothing: the stop step writes every dirty entry and publishes no event.
    • TestTargetsChanged_Reassesses: after SetBillingMode to personal the tracked entry reports difference (a) and its start time.
    • TestPresenceGauges: heartbeat age and mismatch age per active agent; a disabled agent has no series.
  • internal/gateway/app: TestHeartbeatAgentMember_CallsPresence, TestHeartbeatAgentMember_ErrorKeepsConnection (the heartbeat_ack is still sent), TestHeartbeatWithoutAgentMember_NoPresence.
  • internal/wire: TestConfig_PresenceTimeoutAboveHeartbeatInterval (equal values abort the start with both key names).

Integration (PostgreSQL 18 via pgtest, the in-process server and the wstest client as in M1-65 and M2-07, HTTP through the real chain; Owner Alex with step-up creates the agent support-agent through M1-51 and issues its API token with a DPoP key from identitytest; support-agent operates no channel; Sam has read on channel Allgemein of space Support and therefore no agent.read on support-agent; Alex, Sam and the agent hold gateway connections in phase live):

  • TestBR05_ServerPart_ActualDiffersFromDesired (BR-05, server part; desired organization): a heartbeat with organization and key_type: api_key gives billing_mode_matches: true, no warnings and mismatch age 0; a heartbeat with personal and key_type: oauth stores reported_billing_mode = personal at once (read before agent.heartbeat_persist_interval passes), writes one audit entry agent.heartbeat allow with billing_mode_old: organization, billing_mode_new: personal, key_type, method, sends Alex agent.presence_changed with reported_billing_mode: personal; getAgent shows billing_mode_matches: false without warning before billing.mismatch_alert_after and with billing_mode_mismatch after it; a heartbeat back to organization ends the warning and writes a second audit entry.
  • TestBR05_ServerPart_UnknownEvidence (BR-05, unknown): a heartbeat without reported_billing_mode and with key_type: unknown gives the warning billing_evidence_unknown at once and a growing mismatch age; nothing grants or denies anything else (a following authorize of agent.read by Alex is unchanged).
  • TestBR05_ServerPart_ChangeDuringRun (BR-05, change during a run): with details.active_runs naming a run id, a mode change is stored and audited at once; reported_details.active_runs is stored with the next flush as an array of id strings; no run, message or event other than agent.presence_changed is written by the server.
  • TestBR05_ServerPart_DesiredNotSupported: Alex, the only owner, sets support-agent to personal with Alex as credential holder (setAgentBillingMode; I1 holds because nobody else has trigger); a heartbeat without reported_billing_mode and with billing_modes_supported: [organization] gives billing_mode_unsupported; with both modes the warning ends.
  • TestPresenceOfflineAfterTimeout (plan: "offline after agent.presence_timeout"): heartbeats every short interval keep online; after they stop, Alex's connection receives agent.presence_changed {state: offline} once the timeout has passed and not before; getAgent, getSnapshot and a new ready show offline; the row has reported_runtime_state = offline after the next flush; the heartbeat age gauge keeps growing.
  • TestHeartbeatAuditOnActualChange (plan: "audit on a change of the reported mode"): exactly one audit entry per change of mode or class, none for a repeated value; the same changing heartbeat sent to two server instances on one database at the same time yields exactly one audit entry.
  • TestBillingModeMismatchAlarm (plan: "alert after billing.mismatch_alert_after visible in the local stack"): with a short billing.mismatch_alert_after the in-process reader shows bothy.agent.billing_mode.mismatch_age for support-agent above that value after a lasting difference and 0 after the difference ends; together with the rule tests below this proves the chain metric, rule and page route.
  • TestPresenceInSnapshot (plan: "presence in the snapshot"): after a heartbeat, getSnapshot and a new ready of Alex list support-agent with state: online and reported_billing_mode; Sam's snapshot does not list it (no agent.read); no human principal appears in agents; the agent's own ready lists itself.
  • TestPresenceChangedDelivery: Alex and the agent's own connection receive the event, Sam's does not; it has no seq and no permission_version; a connection still catching up does not receive it; the frame decodes into AgentPresenceChangedEvent with unknown members rejected and contains no null.
  • TestSendHeartbeatRightsAndRoute (AZ-P1, AZ-P2, AZ-P13): the route is registered with agent.heartbeat and rl.agent_runtime, agent.presence_changed is a registered ephemeral type; Sam's token and an app token get 403 forbidden with reason: subject_kind_not_allowed; the denials stand aggregated in the audit log; delivering presence events writes no audit entry.
  • TestGatewayHeartbeatEqualsRest: the same bodies through sendHeartbeat and through the gateway member agent lead to equal stored state, audit and events; an invalid details key over the gateway answers error with validation_failed, keeps the connection and still sends heartbeat_ack; Sam's gateway heartbeat with member agent answers error with forbidden; an agent heartbeat without the member leaves presence unchanged.
  • TestSendHeartbeatValidationLeaksNothing (SEC-IN-1; defence for BR-13): bodies carrying a canary string shaped like base64 key material (uppercase letters, /, =) in billing_evidence.method, runtime_version, details.adapter.name and a degraded_reasons entry are rejected with 400; the canary appears in no log record, audit row, event_log row or stored column.
  • TestReportedToolClassImmediateAndOnlyRaises (07 2.11; red team "faking the reported mode"): a heartbeat with reported_tool_class: host while the desired class is chat makes AgentPolicySource.AgentPolicy return the new reported class at once without a change of permission_version; with desired host (setAgentToolClass) a heartbeat reporting chat leaves the effective class host.
  • TestPresenceStartRule: a second server instance started on a database where support-agent is stored online with a fresh last_heartbeat_at keeps it online until agent.presence_timeout after its start, then offline without a heartbeat; with an old last_heartbeat_at it is offline at once.
  • TestPresenceFlushMonotoneAcrossInstances: instance A goes offline for the agent while instance B received and stored a newer heartbeat; A's flush does not overwrite B's row.
  • TestPresenceShutdownFlush: a heartbeat right before shutdown is stored, connections close with 4013, and no agent.presence_changed is sent during shutdown.
  • TestHeartbeatProjectReadiness (world of M2-03: forum Projekt-A bound with the agent Runner as effective operator): a heartbeat of Runner with project_readiness: [{channel_id: Projekt-A, ready: true}] sets project_bindings.ready; repeating it does not call the port again; ready: false resets it; a duplicate channel gives 400 with pointer /project_readiness.

Alarm (Prometheus rule unit tests in infra/observability/prometheus/tests/bothy-alerts.test.yaml, run by the operations checks of M1-04 inside make check; see Scope if they are missing):

  • BothyBillingModeMismatch fires with label bothy_agent_handle and the rendered summary for a gauge series above the default of billing.mismatch_alert_after; it does not fire for a series exactly at it; the existing counter case still fires. The alert-thresholds check finds the final comparison equal to the default of billing.mismatch_alert_after, and the routing check sends page to external-oncall and bothy-ops.

Contract and others: every response and frame of the tests above decodes into its generated type with unknown members rejected. If the contract test of M1-71 is on the base, it covers the newly enabled sendHeartbeat. If services/server/test/canary exists (M1-69), its test TestSEC_T10_ScenarioCoversEveryEnabledOperation fails for sendHeartbeat: add a heartbeat with a canary value in runtime_version to the scenario; allowed sinks are the HeartbeatAck, getAgent and listAgents answers and the column reported_runtime_version; logs, traces, metrics, events and audit stay free of it. No platform-specific code: the tests run on Linux in CI and locally on any system with a Docker-compatible runtime.

Test IDs of the plan row and related: BR-05 (server part): an actual mode unequal to the desired one, an unknown mode and a change during a run are stored, audited, shown as warnings and alarmed, and the heartbeat never lets a run start (the start checks are the adapter's and BR-06 of M2-09). AZ-P1: the new route names agent.heartbeat. AZ-P2: the ephemeral type and the op stay registered. AZ-P5: the snapshot lists presence only for agents the principal may read. AZ-P13: denials are audited, delivery filtering writes no audit.

Commands that must pass, from the repository root:

  • cd services/server && go test -race -count=1 ./internal/agents/... ./internal/gateway/... ./internal/sync/... ./internal/chat/... ./internal/authz/... ./internal/kernel/... ./internal/platform/otel/... ./internal/wire/...
  • make codegen-check, make arch, make register-check, make api-lint, make api-tables, make api-checks, make config-doc-check, make docs
  • make check

Acceptance criteria

  • sendHeartbeat is enabled and registered with agent.heartbeat and rl.agent_runtime; the gateway op heartbeat processes member agent with the same effect; AZ-P1, AZ-P2 and AZ-P13 pass.
  • BR-05 (server part): an actual mode unequal to the desired one, an unknown mode, a desired mode outside the provable modes and a change during a run are stored, audited and shown as the warnings of 10 "Expected and reported"; no heartbeat starts, ends or allows anything.
  • An agent becomes offline once agent.presence_timeout has passed without a heartbeat, and agent.presence_changed reaches exactly the principals with agent.read, never before catchup.complete.
  • Every change of reported_billing_mode or reported_tool_class is stored at once and audited once with old value, new value and method; other fields are stored at most once per agent.heartbeat_persist_interval and at shutdown.
  • bothy.agent.billing_mode.mismatch_age exceeds billing.mismatch_alert_after for a lasting difference, and BothyBillingModeMismatch fires on it in the rule tests with route page.
  • getSnapshot, ready, getAgent and listAgents show the current presence; people never have presence.
  • A reported tool class counts for SR2 at once and can only raise the effective class.
  • docs/api/openapi.yaml, docs/api/asyncapi.yaml, chapters 02, 09, 10 and 18, signals.md, alerting.md, dashboards.md, the READMEs and docs/DECISIONS.md describe what was built.
  • make check is green.

Definition of done

  • Code and tests as specified; doc comment on every public symbol (purpose, parameters, return value, errors, example where useful); comments explain why.
  • READMEs of agents, gateway and sync updated (port Presence, tracker, state rules, persistence, start and shutdown, events, metrics, the agent part of the op heartbeat, presence in the snapshot).
  • Rulings of Design note 17 recorded in docs/DECISIONS.md (one row each with reason); row D-G-280 names the row that ends its presence part.
  • services/server/CHANGELOG.md entry under 'Unreleased'.
  • If docs/security/threat-model.md 5.10 lacks the threat "Adapter reports a false reported mode", add the row with its countermeasures (heartbeat evidence only for display, warning, audit and alarm; I3 in the adapter; BR-06 at run.start; the reported tool class only raises the effective class) and register T-AI-08.
  • An ADR only if you take an architecture decision beyond this brief (MADR 4 as in docs/adr/0000-template-madr-4.md, headings and text in English, next free number, make adr-index).
  • make check green.
  • An independent code review over the diff, then the red team gate (docs/quality/red-team-gate.md, verdict JA or NEIN); on NEIN fix with a test per finding and have the delta checked again. Record both verdicts.
  • Land the change as AGENTS.md ("How work lands on main") describes: a pull request from this branch, squash-merged only when review, red team and make check, ci-ok and the review threads are green; then git worktree remove.

Report

Write to {{REPORT_PATH}}:

  1. First line exactly Verdict: DONE or Verdict: BLOCKED <reason>.
  2. Commits (SHA and subject), changed files with line counts, test results with counts (the red run before the change and the green run after), the summary of make check.
  3. Review and red team verdicts and the findings fixed.
  4. What is not proven (for example the timing margins of the timer and flush tests, presence with cluster.mode = multi beyond the two-instance tests, the alarm in a running local stack beyond the rule tests and the in-process metric reader).
  5. Follow-ups that could not be done in this scope, each with its reason (at least the fallbacks taken for M1-68 or M1-04, if any), and spec gaps found.

Then write {{DONE_PATH}} with exit=0 sha=<sha> or exit=1 <reason>.