Skip to content

M2-47: Terminal relay across several nodes (T6)

Status: ready | Depends on: M2-44 | Milestone: M2

Goal

With cluster.mode = multi several api nodes serve the terminal as one. The node that processes openTerminalSession sends the instruction session.open over NOTIFY ephemeral, only with IDs and only after its transaction committed, to whichever node holds the control connection of the host service; the edge routes both stream paths of a session (/api/v1/terminal/sessions/{session_id}/stream of the device and /api/v1/hosts/sessions/{session_id}/stream of the host service) by consistent hashing over the {session_id} segment to the same api node, where the relay of M2-44 pairs them in memory. A side that lands on another node because the edge lacks the hash rule finds no counterpart within terminal.open_timeout, closes with 4102, is counted in bothy.terminal.relay.unpaired and leaves /readyz ready: a misconfigured edge fails visibly instead of passing bytes through a broker (24 "Multiple nodes", Ruling T6, R-TM-09, R-OP-19, RK-29). The same signal path carries what else the specification asks of several nodes for the terminal: a host transition decided on any node closes the host's connections on every node at once (24 3.1, 3.3), the probe of an older control connection reaches the node that holds it (24 "Second control connection"), and every node knows whether a host is connected anywhere, so 409 host_unavailable behaves as with one node (24 6.1 point 2, 13). The end of a session needs no signal: M2-44's relay learns every committed end from the event-log tail of its own process on every node (24 "Revocation", 02 11.2), and this ticket proves that with two nodes. With cluster.mode = single nothing changes and no NOTIFY is sent. The ticket also ships the example edge with hash routing as a Compose overlay with two api nodes in infra/compose and updates 16 "Terminal: forwarding and host service" (4.4). This is TM-T15 at the server level with the fake host of M2-44; M2-48 runs it again end to end with the real host service in a container. The multi-node relay comes in M2, independent of the rest of the multi-node operation in M7-17 (P-25, OQ-07).

Read first

  • docs/PLAN.md: section 3 "Way of working per ticket"; section 5 (rows "Test doubles", "End to end", "Load, endurance, chaos"); section 8 "M2 Agent platform, extensions and terminal" (Goal, Scope, Out of scope, Exit criteria 8 and 10, Test strategy with "two api nodes with and without hash routing (TM-T15)", Red team (Terminal) with "misrouting across several nodes", Risks), the introduction of "Block terminal" and the rows M2-38 to M2-44 (what exists), M2-47 (this ticket, the contract), M2-45, M2-46 and M2-48 (who builds what next); row M7-17 (the rest of the multi-node operation); section 19 P-09, P-10, P-11, P-20, P-25.
  • Dependencies: the header names M2-44. Through it this ticket builds on M2-43, M2-42, M2-41, M2-40, M2-39, M2-38 and on the M1 tickets M1-07, M1-08, M1-10, M1-11, M1-12, M1-13, M1-14, M1-16, M1-17, M1-18, M1-23, M1-24, M1-28, M1-32, M1-33, M1-34, M1-35, M1-44, M1-48 and M1-49 (all implied, so not in the header). D-W-102 struck no dependency from the column of this ticket (it always named only M2-44); it struck M2-40 from the column of M2-44 and M2-39 from the column of M2-43; both lie in this ticket's closure and stay in this list. Because M2-43 depends on M1-67 (D-W-105), M1-64, M1-65, M1-66 and M1-67 with their own closure, among them M1-21 (trusted proxies), are in this ticket's closure as well. Not in the closure, read for coordination only: M1-04 (operations checks), M1-68 (metric registry, signal-check), M1-70 (visibility lint), M1-73 (compose smoke) and M2-46 (revocation, limits, the other terminal metrics).
  • docs/spec/24-terminal.md: 2.1 "Overview" (diagram with the edge and hash routing), 3.1 "States" (every transition out of active ends all sessions), 3.3 "Re-pairing" (closes the host's connections), 3.4 "Second control connection" (with cluster.mode = multi over NOTIFY ephemeral to the node of the older connection), 5.3 "Terminal session token" (the host service checks the token with the JWKS from hello), 5.4 "Frames" (the relay sees only length and time), 6.1 "Opening" points 2 to 7, 6.2 "Detach and reattach", 6.3 "Limits", 8.3 "Revocation", 9 "Multiple nodes" (all), 10.1 to 10.4, 13 "Errors" (close codes 4100 to 4114 and their channels), 14 "Tests" (TM-T15; TM-T16 and TM-T23 for context), 16 (R-TM-09, R-TM-13, R-TM-16), 17 (terminal.open_timeout, terminal.idle_timeout, terminal.idle_grace, terminal.host_heartbeat_interval, terminal.host_heartbeat_miss_limit, terminal.host_duplicate_probe_timeout).
  • docs/spec/16-operations.md: 4.1 "Basic rules" rule 1, 4.2 "Two listeners" (/readyz), 4.3 "Edge and identity" (server.trusted_proxies, profile direct), 4.4 "Terminal: forwarding and host service" ("Edge and proxy" points 1 to 3), 15 R-OP-18 and R-OP-19.
  • docs/spec/02-architecture.md: 4.3 (row terminal), 4.4, 5 "Layers within a module" (platform only in adapters), 11.2 "Log tail per api process", 11.6 "Ephemeral events", 11.7 "Cost of serialization", 12.3 (row terminal.sweep), 14.4 (row LISTEN connection), 15.1, 15.2 "Health", 15.3 "Shutdown" (api step 4), 16 "Scaling path" (level 2 and "Multiple nodes in detail"), 19 R-AR-10, 22 (cluster.mode, database.listen_reconnect_backoff_cap).
  • docs/spec/03-data-model.md: tables hosts and terminal_sessions; 6.6 "Notifying the gateway processes" (the channel ephemeral carries only IDs and versions, among them host and terminal session, and is off with one node). docs/spec/04-sync-protocol.md: 6.4 (last paragraph: terminal channels have their own codes), 12 point 3, R-SY-25. docs/spec/05-actions.md: 4.14 "Hosts and terminal" (host.connect, terminal.open, terminal.attach, terminal.stream, terminal.close, terminal.terminate), 6 (terminal_session.changed, host.updated, host.presence_changed).
  • docs/spec/07-permissions.md 2.12 "Hosts and the terminal capability", SR10, SR11, SR12; docs/spec/08-identity.md 6.3 "Fresh step-up for the terminal" and ID-T28; docs/spec/19-security.md SEC-TERM-1, SEC-TERM-4, SEC-TERM-5, SEC-LOG-3, SEC-LOG-5; docs/spec/17-observability.md 2 "Basic rules".
  • docs/api/asyncapi.yaml: channels terminalStream, hostControl and hostData with their x-close-codes; messages host.hello, host.heartbeat, host.report, host.session.open, host.session.close, host.session.started, host.session.ended. docs/api/openapi.yaml: operations openTerminalSession, attachTerminalSession, closeTerminalSession, setHostState, issueHostPairingCode; schemas HostSessionOpen, HostSessionClose, HostSessionEnded, TerminalCloseReason, TerminalCloseDetail, Problem (x-problem-codes host_unavailable, terminal_denied, step_up_required).
  • docs/observability/signals.md 2 "Cardinality", 4.3a "Terminal" (bothy.terminal.relay.unpaired with bothy.terminal.side), 5.3 "Attributes in spans and logs", 7 "Logs"; infra/otel/collector.yaml (redaction/allowlist.allowed_keys).
  • docs/operations/configuration.md: 4 (the keys above, cluster.mode, database.listen_reconnect_backoff_cap, database.statement_timeout, sync.heartbeat_interval, secrets.signing_key_dir, server.trusted_proxies), 9 "Compose variables", 10 "Maintenance" (a key is never reinterpreted).
  • infra/compose/compose.yaml, infra/compose/.env.example, infra/compose/README.md, docs/operations/install-compose.md 5 "Edge".
  • ADR 0066 (writable key store), ADR 0067 (JOSE profile), ADR 0068 (section "Confirmation").
  • docs/DECISIONS.md: D-T6, D-T13, D-N06, D-L23, D-OP-03, D-W-100, D-W-101, D-W-102; section 10 for the format of new rows.
  • docs/security/threat-model.md 5.3 "Network edge and proxy", 5.16 "Terminal", 6.7 (T-TM-01, T-TM-02, T-TM-05, T-TM-11), 9 "Maintenance rule". docs/RISKS.md RK-29, RK-37. docs/OPEN-QUESTIONS.md OQ-07 (multi-node in M7, the terminal relay with this ticket) and OQ-24 (terminal only over a private network, default no); OQ-03 and OQ-04 are not touched here.
  • Briefs: docs/briefs/M1-11-lifecycle-health.md (lifecycle.Component, lifecycle.StopStep, lifecycle.PhaseEventTail, Runner.Fail), M1-12-postgres-migrations.md (pgtest, testcontainers-go with Ryuk off and loopback ports), M1-13-unit-of-work.md (txn.UnitOfWork.Do, phase work, pg.Tx from the context), M1-16-event-log.md (pg.DialDedicated, the tail's LISTEN loop as the pattern), M1-07-config-schema-loader.md and M1-08-config-reference.md (keys enter the schema with their first consumer, configkeys_*.go, chapter constants), M1-10-telemetry-foundation.md (internal/platform/otel, otel/attr, TestAttrKeysEqualCollectorAllowlist), M1-23-server-image.md, M1-32-signing-keys.md (key store, purpose terminal, JWKS), and the briefs of M2-38 to M2-44 where they exist in docs/briefs/; otherwise their plan rows.
  • Code on the base branch, read with git show before changing anything: internal/terminal (port, domain, app, adapters/postgres, adapters/httpapi, adapters/wsrelay), test/terminal/fakehost (M2-44) and the device client of M2-44's relay tests, internal/platform/pg and pgtest, internal/platform/lifecycle, internal/platform/otel, internal/kernel/ids, internal/kernel/clock, internal/wire, architecture/modules.yaml.

Fixed requirements that apply

Owner's fixed build rules (apply verbatim):

  1. No MVP. New from scratch and directly in the final state; stages are only the order.
  2. Strictly modular: clear module and service boundaries, dependencies only through defined interfaces, no cross access.
  3. Documentation in code: every public module, class, function and interface with a docstring/doc comment (purpose, parameters, return value, errors, example where useful); comments explain the why, not the what. API documentation is generated from code (OpenAPI from the routes, dartdoc, Python API docs in MkDocs).
  4. Documentation in the repo: README per module/service, ADR for every architecture decision, CHANGELOG per release, keep the documentation site current. A change without matching documentation is not done.
  5. Quality gates (SonarQube, linters, type checking, security scanners, test coverage) block the merge.

Rules for every brief:

  1. Test first: write the failing test, run it and keep the red output for the report, then make it green, then clean up.
  2. No en dash and no em dash in any file you write; the text-rules pre-commit hook rejects them. Use a hyphen or rephrase.
  3. White label: examples and test data use example.com and the people Alex, Sam and Robin; no company, host or personal names, no paths under a home directory.
  4. Limits, windows and thresholds only through named configuration keys listed in docs/operations/configuration.md; code comments and docs name the key, never the number.
  5. Specification first: a change to behaviour, API, event, right or configuration updates docs/spec/ or docs/api/ in the same change. The core contract (docs/spec/00-glossary.md, 03-data-model.md, 04-sync-protocol.md, 05-actions.md) is not edited by this ticket.

Product requirements implemented here:

  1. With cluster.mode = multi the node that processes terminal.open sends the instruction session.open over NOTIFY ephemeral, only with IDs, to the node of the control connection; the receiving node reads everything else from the database (24 9 point 1, 03 6.6, 02 11.6, R-AR-10, R-SY-25, D-T6).
  2. The host service opens one data connection per session on /api/v1/hosts/sessions/{session_id}/stream, the device connects to /api/v1/terminal/sessions/{session_id}/stream, and the edge routes both by consistent hashing over the {session_id} segment to the same api node, where the relay connects the two sides in memory; no broker, and no byte of a stream ever goes through the database, the event log or NOTIFY (24 9 points 2 and 3, 16 4.4 "Edge and proxy" point 2, R-TM-09, R-OP-19, 24 10.2).
  3. A side that lands on another node finds no counterpart within terminal.open_timeout and closes with 4102; the node counts this in bothy.terminal.relay.unpaired (M2-44's counter, D-W-105) and keeps /readyz ready (24 9 point 4, signals.md 4.3a, TM-T15, PLAN row M2-47).
  4. The channel ephemeral is active only with cluster.mode = multi; with one node nothing of this ticket runs (03 6.6, 02 11.6, R-AR-10).
  5. With cluster.mode = multi the probe of an older control connection reaches the node of that connection over NOTIFY ephemeral (24 3.4), and a transition of a host decided on any node ends the affected connections on every node at once, fail closed (24 3.1, 3.3, D-L23); the end of a session reaches every node through the event-log tail of M2-44 (its Design note 11; 24 8.3, 02 11.2), which this ticket proves with two nodes and does not duplicate with a signal.
  6. The edge forwards WebSocket upgrades for /api/v1/terminal/sessions/{session_id}/stream, /api/v1/hosts/connect and /api/v1/hosts/sessions/{session_id}/stream with an idle timeout that never cuts a quiet terminal, does not buffer these connections and writes no content to logs (16 4.4 "Edge and proxy" points 1 to 3); published ports bind 127.0.0.1 of the host only (16 4.1 rule 1).
  7. Telemetry only as the catalogue allows: this ticket registers no instrument and no attribute; it asserts M2-44's bothy.terminal.relay.unpaired (D-W-105) with the closed attribute set bothy.terminal.side (client, host) and uses that attribute, which M2-44 registers, in its log records; no IDs of people, no host names, no content (signals.md 2 and 4.3a, 24 10.4, SEC-TERM-1, SEC-LOG-3, SEC-LOG-5).
  8. Every signal carries only a kind and IDs: never a session token, jti, profile, name, path, address or byte of a stream (03 6.6, 02 11.6, SEC-TERM-1).

Terminal security requirements (D-W-101):

  1. Rights per machine: the additional right terminal on the target host, SR10, SR11 with I6 and SR12 (07 "Hosts and the terminal capability"). M2-40 builds them as decisions of authorize and M2-44 applies them at terminal.open and terminal.attach (AZ-M54 to AZ-M61, TM-T08, TM-T09, TM-T28 (a)); this ticket adds no right, action or decision. What it enforces: a decision stays bound to the node and the transaction that made it. A signal carries only IDs and causes nothing the committed rows do not allow; the control node sends session.open only for a committed session in opening of an active host, so a rolled-back or forged signal opens nothing (TestCrossNode_SessionOpenOnlyAfterCommit, TestCrossNode_ForgedSignalOpensNothing), and an end of a session decided on any node closes its connections on every node (TestCrossNode_CloseReachesRelayAndControl).
  2. Fresh step-up: device key in the apps, only passkey in the console (08 6.3, ID-T28). M2-40 builds step-up fresh with its consumption in the opening transaction (ID-T28 without the console part), M2-44 TM-T07 without the console part and TM-T12, M6-18 the console with passkey only, M4-36 and M5-12 the step-up key in the apps. What this ticket enforces: the consumption lives in the database, not in a node; two opens with one confirmation on two nodes give exactly one session and one 401 step_up_required (TestCrossNode_StepUpConsumedOnce).
  3. Network: there is no public inbound path to the terminal; the host service opens only outbound connections (R-TM-01); an installation with the profile tailscale reaches the server only over the tailnet with a policy only for port 443 and without Funnel (D-OP-03, docs/operations/tailscale.md); the product does not require Tailscale; whether the terminal must be reachable only over a private network is owner question OQ-24 (default no). This ticket adds no inbound path: the example edge publishes only on 127.0.0.1 of the host behind the operator's TLS edge, the api nodes publish no port, and the host service still reaches the server only outbound through the same edge (TestComposeMultiNodeOverlay); the overlay works only with the profile direct, because tailscale serve has no hash rule, and with the profile tailscale it adds nothing (TestComposeMultiNodeOverlay, case tailscale). On an answer "yes" to OQ-24, M2-43 and M2-44 change and the overlay follows them.
  4. Audit without content: M2-46 owns audit and telemetry without content (TM-T06), M2-44 the relay that records only length, time and session ID. What this ticket enforces: signals carry only IDs (TestCrossNode_SignalsCarryOnlyIDs, TestSignalCodec); the end of an unpaired session writes exactly one terminal.terminate entry with codes only, also when two nodes time out (TestTMT15_WithoutHashRouting); the example edge writes no access log and no request line (TestEdgeExample_Config); M2-44's metric has only the attribute bothy.terminal.side (TestRelayUnpairedAttributeSet).
  5. Idle timeout and maximum duration (24 "Limits"): M2-45 measures idle on the host, M2-46 the server side and the job terminal.sweep (TM-T11). What this ticket enforces: an end committed on any node, also in a worker transaction, closes the relay connections on the node that holds them through M2-44's event-log tail, which this ticket proves with two nodes (TestCrossNode_CloseReachesRelayAndControl, subtest worker_end); and the example edge never cuts a session before the server would end it, because its timeout on the stream paths is terminal.idle_timeout plus terminal.idle_grace (TestEdgeExample_Config).
  6. Service user and session users without administrator rights (T12, HD rules): M2-42 builds install and doctor with HD1 to HD11 (TM-T13, TM-T22, TM-T24), M2-45 runs every PTY in the helper of the session user. Not touched here; the example edge runs as a non-root user, read-only, without capabilities and with no-new-privileges, like the server containers (infra/compose/README.md "Security rules in this file", TestComposeMultiNodeOverlay, TestEdgeExample_Config).
  7. Threats of threat model 6.7 that concern this ticket, each with its test: T-TM-01 (content readable in the server) by a relay that pairs both sides on one node and never moves a byte across nodes, with signals of IDs only (TM-T15 tests, TestCrossNode_SignalsCarryOnlyIDs; TM-T06 by M2-46); T-TM-02 (replay and token swap at the relay) by a misrouted side that never receives a byte of the other side (TestTMT15_WithoutHashRouting; TM-T03 to TM-T05 by M2-45); T-TM-05 (copied keys, a second control connection) by the probe across nodes and the probation of a new connection (TestCrossNode_DuplicateControlConnection; TM-T23 on one node by M2-43); T-TM-11 (session stays after revocation) by ends that reach every node through the event-log tail, also while the listener of this ticket is lost, and the reconciliation of host transitions after a lost listener (TestCrossNode_CloseReachesRelayAndControl, TestCrossNode_ListenerLossReconciles; TM-T10 by M2-46 and M2-48); and the new row T-TM-15 (misrouting and lost signals across nodes, design note 15) by TM-T15. Not touched here: T-TM-03 and T-TM-09 (M2-40, M2-44), T-TM-04, T-TM-12, T-TM-13 and T-TM-14 (M2-42, M2-45), T-TM-06 (M2-41), T-TM-07 (M6-18), T-TM-08 (M2-40, M2-42, M2-45), T-TM-10 (M4-35).

Scope

All paths under services/server/ unless noted. Names of M2-38 to M2-44 below are the expected ones; where the merged code names them differently, follow the code (Boundaries).

Create:

  • internal/terminal/domain/cluster.go (pure, standard library and internal/kernel only): type SignalKind string with SignalSessionOpen = "terminal.session_open", SignalHostClose = "terminal.host_close", SignalHostPresent = "terminal.host_present", SignalHostGone = "terminal.host_gone", SignalSessionPaired = "terminal.session_paired"; type Signal struct { Kind SignalKind; HostID, SessionID, ConnID ids.ID }; EncodeSignal(s Signal) ([]byte, error) and DecodeSignal(payload []byte) (s Signal, ours bool, err error) (design note 3); OlderConnection(a, b ids.ID) bool (design note 7); type PresenceView with Seen(host, conn ids.ID, at time.Time), Gone(host, conn ids.ID) and Connected(host ids.ID, now time.Time, window time.Duration) bool (design note 6); ProbationEnd(firstPresent time.Time, probe time.Duration) time.Time (design note 7); type Side string with SideClient = "client", SideHost = "host", LoneSideOutcome(side Side, state string) LoneOutcome and LoneDeadline(identifiedAt time.Time, openTimeout time.Duration) time.Time (design note 8).
  • internal/terminal/app/signals.go: outgoing port Signals and inbound handler interface SignalHandler (design note 4) and LocalSignals, the implementation for cluster.mode = single.
  • internal/terminal/adapters/postgres/queries/cluster.sql (sqlc): NotifyEphemeral (SELECT pg_notify('ephemeral', sqlc.arg(payload)::text)) and HostStatesByIDs (state and disabled_reason by ID list, ordered by id) unless M2-39 or M2-44 already have an equal read.
  • internal/terminal/adapters/postgres/signals.go: NotifySignals, the implementation of Signals for cluster.mode = multi; internal/terminal/adapters/postgres/ephemeral_listener.go: EphemeralListener (a lifecycle.Component named terminal_ephemeral, design note 5).
  • internal/terminal/adapters/wsrelay/cluster.go: the hub's HandleSignal, the presence view, the probe across nodes, the probation and the reconciliation (design notes 4 to 9).
  • internal/wire/configkeys_cluster.go: clusterConfigKeys() with cluster.mode (enum single, multi, default from 02 22, chapter02), only if the base branch does not register the key yet.
  • test/terminal/multinode/: main_test.go (pgtest.Main), harness_test.go (two in-process nodes, design note 13), testedge_test.go (the in-process test edge) and the integration tests of the section "Tests".
  • test/edge/: main_test.go, edge_example_test.go, compose_overlay_test.go (design notes 11 and 12).
  • Repository root: infra/compose/compose.multi-node.yaml and infra/compose/edge/nginx.conf (design notes 11 and 12).

Change:

  • internal/terminal/app: every use case of M2-39 to M2-44 that acts on a live connection after its commit other than through the event-log tail (open, attach, host state changes, re-pairing, the duplicate verdict) stages its signal in the unit and hands it over after the commit through Signals instead of calling the relay directly (M2-44's RelayControl.Expect and ControlLinks.OpenSession after open and attach move into the handler of terminal.session_open, design note 3); the check "host connected" of open and attach asks the presence port of design note 6. Ends of sessions (close by the person or an owner, the unpaired end, the end with host_restarted after identify, the storing of session.ended, and the ends of M2-46 and terminal.sweep) keep M2-44's path through the event-log tail and stage no signal (design note 3).
  • internal/terminal/adapters/wsrelay: a connection ID (UUIDv7 from an ids.Generator on the node's clock) per accepted control connection; the presence view; the rules of the lone side (design note 8) where M2-44 differs; HandleSignal.
  • internal/wire: choose LocalSignals or NotifySignals by cluster.mode; in roles api and all with multi start EphemeralListener after the HTTP listeners and register its stop step in lifecycle.PhaseEventTail; role worker gets NotifySignals without listener; register cluster.mode if missing; regenerate internal/wire/testdata/txn_registrations.golden only if it changes.
  • docs/api/asyncapi.yaml: 4102 terminal_host_unavailable in channels.hostData.x-close-codes; the client_action of 4102 in terminalStream after design note 8; then make api-lint and make api-checks.
  • docs/spec/24-terminal.md, docs/spec/16-operations.md, docs/spec/02-architecture.md, docs/security/threat-model.md, docs/adr/0068-terminal-through-a-host-service-with-noise-stream-and-invariant-i6.md ("Confirmation"), docs/operations/install-compose.md (new 5.3), docs/operations/configuration.md (section 9, one row) as in design note 15; make adr-index.
  • Repository root: infra/compose/README.md (files, overlay, checks), infra/compose/.env.example (BOTHY_NGINX_VERSION).
  • internal/terminal/README.md, services/server/CHANGELOG.md and the root CHANGELOG.md (overlay of the Compose bundle) under "Unreleased", docs/DECISIONS.md section 10 rows D-M2-47-1 to D-M2-47-11.
  • tools/lint/sql_visibility.yaml: only if it exists (M1-70 merged), the reads of design notes 4 and 9 with the reason "Relay reads the state of its own connections".

This ticket creates no migration and no new top-level make target.

Out of scope

  • The relay on one node, session tokens, openTerminalSession, attachTerminalSession, listTerminalSessions, getTerminalSession, closeTerminalSession, terminal_session.changed, the close path of every session end through the event-log tail, the counter bothy.terminal.relay.unpaired (D-W-105) with its attribute bothy.terminal.side, the fake host and the single-node cases TM-T07 and TM-T12 (M2-44); the control connection, hello, identify, report, heartbeats, close codes 4110 to 4114, the local probe of TM-T23 (M2-43).
  • Handshake checks, PTY, profiles and replay rejection on the host (M2-45); revocation subscribers, limits, rl.terminal and rl.host, terminal.sweep, the metrics bothy.terminal.sessions.active, bothy.terminal.sessions.closed and bothy.terminal.hosts.connected, TM-T06, TM-T10, TM-T11 (M2-46). Their ends reach the connections through M2-44's event-log tail like every end; this ticket builds nothing for them.
  • The conformance and chaos suite with the real host service in a container, TM-T15 end to end through the example edge, TM-T16 (M2-48).
  • NOTIFY ephemeral for gateway, runs and agents (typing.started, run.updated, agent.presence_changed, 4017 across nodes), fencing signals of agent consumers, host.presence_changed to clients on another node, a shared key store for every deployment target, rate limits across nodes, perf.fanout_cross_node_p99 (M7-17). Rate limits stay per node (02 16).
  • Starting the overlay (docker compose up), its support in install.sh (M7-29), ingress examples for Helm, other proxies (the overlay is one example; the rule stands in 16 4.4).
  • A terminal reachable only over a private network (OQ-24); content recording (OQ-04).
  • Changes to the core contract.

Boundaries

  • Work in a dedicated git worktree on the branch feat/M2-47-terminal-multi-node; one writer per worktree.
  • Conventional Commits with the module as scope (for example feat(terminal): ..., docs(compose): ...) and a DCO sign-off on every commit (git commit -s); squash into meaningful commits before the merge (docs/quality/push-discipline.md).
  • Never push to main, never tag and never create remote repositories; the only remote steps are the branch push and the pull request of AGENTS.md.
  • Do not modify files outside the scope above, except the CHANGELOG, README and docs/DECISIONS.md entries named in this brief.
  • Follow the host rules supplied by the launcher (ports, machine rules).
  • Tests bind only 127.0.0.1 and port 0; PostgreSQL runs only as a throwaway container on a random port through pgtest (P-09, P-10). Without a Docker-compatible runtime the integration tests fail; they never skip.
  • Stop only processes you started, by PID. Temporary directories only via mktemp -d, removed at the end.
  • Never read or print secrets. No network calls except package downloads for pinned tools and Go modules.
  • Never lower a threshold, disable a gate or delete a test.
  • If the core contract lacks something this ticket needs, stop with Verdict: BLOCKED <reason> instead of building a deviation (P-11).
  • This brief was written before milestone M1 closed (D-W-100). Where the merged code of an M1 ticket names a package, type, function, file or make target differently from this brief, follow the code and list each difference in the report; where it behaves differently from the specification, stop with Verdict: BLOCKED <reason>.
  • The same holds for M2-38 to M2-44: their merged names win over the expected names here; list each difference. Where the merged relay of M2-44 differs from design note 8 in a point that 24 9 and 13 decide (the close code of a lone side, the timer of the host side), change it here and list it; any other difference of M2-38 to M2-44 from the specification: stop with Verdict: BLOCKED <reason>.
  • Containers only from pinned images: PostgreSQL through pgtest and the pinned nginx image of the overlay for the edge tests, with Ryuk disabled, ports published only to 127.0.0.1 with random host ports, label org.bothy.test=edge, removed at the end. Pulling these images counts as a download of a pinned tool. docker compose only with the subcommand config, never up, run or pull.
  • No real host service, no real device, no third-party service; nothing in this ticket costs money.
  • Time only through kernel/clock (Now) and the timer source of M2-44's relay; no test sleeps for the default value of a window (design note 13).
  • Session tokens, jti values, keys and stream bytes never appear in logs, signals, audit details, problem bodies or test failure messages (print a hash prefix if needed).

Design notes

  1. Scope of the ticket (Ruling for docs/DECISIONS.md, D-M2-47-1). The plan row names session.open over NOTIFY ephemeral and the hash rule of the edge. With several nodes the specification needs the same path in three more places, and no other ticket builds them (M7-17 covers gateway, runs and agents only): the probe of an older control connection "through NOTIFY ephemeral to the node of the older connection" (24 3.4); the closing of a host's connections at a transition or re-pairing (24 3.1, 3.3); and the check "host connected" behind 409 host_unavailable (24 6.1 point 2, 13). They belong here, because without them cluster.mode = multi would keep the control connection of a disabled or re-paired host open on another node, hide a copied key and answer 409 host_unavailable by chance of the node. The immediate end of sessions with 4100 and the instruction to the host service (24 8.3) needs no signal: every end writes terminal_session.changed in its transaction, and the event-log tail of every api node hands it to M2-44's relay observer (M2-44 Design note 11; 02 11.2, 16), so this ticket only proves it with two nodes. The code stays node-neutral (OQ-07); with cluster.mode = single none of it runs.

  2. Mode and roles (Ruling for docs/DECISIONS.md, D-M2-47-2). cluster.mode is the enum single or multi of 02 22 (default from that table, place Server). Register it in internal/wire/configkeys_cluster.go with chapter02 if the base branch lacks it (keys enter with their first consumer, M1-07); then make config-doc-check.

    • single: LocalSignals, no listener, no NOTIFY, no presence view, no probation; the behaviour and the tests of M2-43 and M2-44 stay unchanged.
    • multi, roles api and all: NotifySignals and one EphemeralListener per process.
    • multi, role worker: NotifySignals for a unit of the Scope it commits, no listener, because it holds no connection; the ends it commits (terminal.sweep, expiries) need no signal (design note 3).

  3. Signals and payload (Ruling for docs/DECISIONS.md, D-M2-47-3). Channel ephemeral (03 6.6). The payload is compact JSON with exactly the fields of its kind: kind, host_id, and per kind session_id or conn_id, each a UUID string. No other field, never null. Kinds, who sends them and what the receivers do:

    • terminal.session_open (host_id, session_id): staged by every unit after which M2-44 sends session.open (the open; the attach too, if M2-44 sends one there). Receivers: every api node loads the session through the app and, for a session of that host that is not closed, registers the expectation locally through M2-44's RelayControl.Expect (M2-44 Design note 7, with the expected device_id of the row and the node's receipt time of the signal as the grant time of its deadline), which closes an older device side of the session held on that node with 4103 and an older unpaired host side with 1000; so the node the edge hashes the session to holds the expectation, whichever node served the REST call (M2-44 Design note 9 closes a host data connection without an expectation with 1000). The deadline of such an expectation decides by the session state read from the database (design note 8), not by a local started mark, because session.started arrives only on the node of the control connection; an expectation without sides therefore ends only a session still in opening, through the compare-and-swap of design note 10. Ruling for docs/DECISIONS.md: on a host whose last report says require_local_confirm = true (hosts.reported_require_local_confirm), an expectation without sides that received terminal.session_paired for the session after its grant time only removes itself at its deadline; without that signal it ends the opening at its deadline like every other expectation (open_failed with open_timeout, compare-and-swap of design note 10). Reason: a pairing on another node meets the opening deadline there, and the host service then bounds start and consent itself and ends a refused or unanswered opening with session.ended (M2-44 Design note 7, M2-45 Design note 15 d); the server does not know the host's host_service.local_confirm_timeout, so only the fact of the pairing may spare the opening, and an opening that no node ever paired still ends after terminal.open_timeout (24 6.1 step 6). A node that holds a side keeps M2-44's rule. Only a node that holds an eligible control connection of the host (one that sent its first report, design note 6) additionally sends session.open, only for a session in opening of an active host, with M2-44's builder of HostSessionOpen (session_id, data_path, profile, expires_at); it drops everything else. A connection in probation waits with the session.open, not with the expectation (design note 7). With cluster.mode = single, LocalSignals.Committed hands the signal to the same handler on the one node, which does both, so M2-44's order (RelayControl.Expect, then ControlLinks.OpenSession) stays.
    • terminal.host_close (host_id): staged by every transition out of active (setHostState), by a re-pairing (issueHostPairingCode on a paired host) and by the duplicate verdict of design note 7. Receivers: every node closes the host's control and data connections with the code M2-43's single-node path uses for the host's state read after the signal (expected: 4110 after a re-pairing, 4112 for disabled_reason = duplicate_connection, 4111 for disabled by an owner and for retired).
    • terminal.host_present (host_id, conn_id) and terminal.host_gone (host_id, conn_id): sent outside any unit by the node of a control connection (design notes 6 and 7).
    • terminal.session_paired (host_id, session_id): sent outside any unit by the node whose relay pairs both sides of a session in opening on a host with require_local_confirm = true; every other node marks its side-less expectation of that session as paired (only IDs, R-SY-25).

    The end of a session has no kind. Every end, from any module, process or node (close by the person or an owner, the unpaired end, host transitions, re-pairing, host_restarted, a stored session.ended, the ends of M2-46 and terminal.sweep), writes terminal_session.changed in its transaction; the event-log tail of every api process reads the committed rows of event_log (woken by NOTIFY event_log, which is on in every mode, 03 6.6) and hands each closed session to M2-44's relay observer, which closes the device stream and the host data connection held there and calls ControlLinks.CloseSession on the node of the control connection (M2-44 Design note 11, 02 11.2). Reason: that path already reaches every api process in both modes, also for an end committed in a worker; a second path through NOTIFY ephemeral would act twice and would not reach the api process from a worker with cluster.mode = single.

    Decoding is strict: unknown fields, missing or extra IDs for the kind, a non-UUID value or an unknown kind starting with terminal. drop the payload with one log record (design note 14); a kind outside the prefix terminal. is ignored without a log, because M7-17 adds its own kinds to the same channel. The longest encoded signal stays below the payload limit of PostgreSQL NOTIFY; the limit is a protocol constant in code with its source, not a configuration key. The kinds are internal signals of the module, not event types of 05: no entry in docs/spec/actions.yaml, no message in asyncapi.yaml, nothing in the event log. Ruling: the receiver always reads state from the database, so signals that arrive late, twice or out of order change nothing wrong.

  4. Port Signals and the handler (Ruling for docs/DECISIONS.md, D-M2-47-4). In internal/terminal/app:

    • type Signals interface { Stage(ctx context.Context, s domain.Signal) error; Committed(ctx context.Context, s domain.Signal); Send(ctx context.Context, s domain.Signal) error }. Stage runs inside the unit of work in phase work; Committed runs after Do returned nil; Send runs outside any unit.
    • type SignalHandler interface { HandleSignal(ctx context.Context, s domain.Signal) }, implemented by the wsrelay hub. It never blocks the caller: frames go to the writer of the connection, and every database read uses a context with database.statement_timeout.
    • LocalSignals (single): Stage does nothing, Committed calls the local handler (for terminal.session_open it registers the expectation and sends session.open, design note 3), Send does nothing.
    • NotifySignals (multi): Stage executes NotifyEphemeral on the unit's transaction, so PostgreSQL delivers the notification only after the commit and never after a rollback; Committed does nothing, because every node, the sender included, receives the notification through its listener; Send executes NotifyEphemeral through the pool in its own statement.
    • Every use case of the Scope ("Change") calls Stage in its unit and Committed after it; no use case calls the relay directly after a commit any more. One handler serves both modes, so the single-node behaviour is the multi-node behaviour with a local transport. Reason: a local shortcut on the sending node would act twice (locally and again through its own listener) or need a second code path.

  5. Listener (Ruling for docs/DECISIONS.md, D-M2-47-5). EphemeralListener opens its own connection with pg.DialDedicated(ctx, "bothy-terminal-ephemeral"), executes LISTEN ephemeral, reads notifications in order on one goroutine, decodes them (design note 3) and hands each one to SignalHandler. On the loss of the connection it logs terminal ephemeral listener lost once, tries again at once and then every database.listen_reconnect_backoff_cap (the largest interval 02 22 allows for a LISTEN reconnect; no new key), runs the reconciliation of design note 9 after LISTEN succeeded again and logs terminal ephemeral listener restored. The listener is a soft dependency like the tail's LISTEN (02 14.4): no readiness check, /readyz unchanged. Stop step terminal_ephemeral in lifecycle.PhaseEventTail (02 15.3 api step 4, "Stop the log tail and LISTEN"); an unexpected end of its goroutine (a bug, not a lost connection) goes to lifecycle.Runner.Fail. One terminal listener per process; M7-17 may merge it with its own listener of the same channel, which the namespaced kinds allow; say so in the terminal README.

  6. Presence across nodes (Ruling for docs/DECISIONS.md, D-M2-47-6). "Host connected" (24 6.1 point 2, 13) needs to know whether any node holds an eligible control connection; 03 has no column for it, so the view comes from signals.

    • The node of a control connection sends terminal.host_present after the connection's first report and again after each heartbeat it receives from the host service, and terminal.host_gone when the connection closes.
    • Each node keeps a domain.PresenceView of remote entries in memory. A remote entry counts while no more than terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit has passed since its last terminal.host_present (the silence after which the holder closes the control connection itself, M2-43); terminal.host_gone removes only its own conn_id.
    • With multi, "host connected" for openTerminalSession and attachTerminalSession is an eligible local control connection or a counting remote entry; otherwise 409 host_unavailable, as on one node.
    • A remote entry of a node that died without terminal.host_gone counts at most for the window above; an open it lets pass ends open_failed with close_detail = open_timeout after terminal.open_timeout. After a start or a listener reconnect the view is complete within one terminal.host_heartbeat_interval; until then an open may get 409 host_unavailable for a host connected elsewhere, which the next attempt corrects.

  7. Second control connection across nodes (24 3.4; Ruling for docs/DECISIONS.md, D-M2-47-7). Each node gives every accepted control connection a conn_id from an ids.Generator on its clock (UUIDv7, ordered by the accept time of its node, kernel/ids). a. A node ignores terminal.host_present of its own connections; a second connection of a host on the same node stays M2-43's local case. b. A node that holds a control connection d of the same host with OlderConnection(d, c) for an announced c starts one probe of d with M2-43's probe (WebSocket ping, terminal.host_duplicate_probe_timeout); at most one probe per connection at a time. No answer: close d with 4009 as M2-43 does and send terminal.host_gone. Answer: run M2-43's duplicate verdict in one unit (host disabled with disabled_reason = duplicate_connection, sessions end host_disabled, audit host.connect with code duplicate_connection, host.updated to all owners), which stages terminal.host_close (the sessions' ends reach the relays through the event-log tail, design note 3); every node then closes the control connections of the host with 4112. c. A node that holds a newer connection d sends terminal.host_present for d at once, so that the node of the older c probes it after rule b. The periodic terminal.host_present of design note 6 repeats this after a lost notification. d. Probation: with multi a control connection receives no session.open before ProbationEnd (twice terminal.host_duplicate_probe_timeout after its first terminal.host_present: the other node's probe window plus the same again for notification and commit). A terminal.session_open for its host during probation waits and is checked again at the end against the database (session still opening, host still active, connection still open); the session's own terminal.open_timeout keeps running. Reason: otherwise a host service with copied keys on another node could get session.open and finish the handshake before the probe decides (T-TM-05). e. The order of two connections on two nodes rests on the clocks of the nodes; servers run with synchronized clocks, and only overlapping connections of one host are compared. Write this into 24 3.4 and the README.

  8. The lone side (24 9 point 4, 6.1 point 6, 13; Ruling for docs/DECISIONS.md, D-M2-47-8). The same rules on one node and on several; this ticket makes them exact, and the counts go to M2-44's counter bothy.terminal.relay.unpaired (D-W-105). a. A device stream connection whose identify was accepted and that has no host data connection of its session on this node within terminal.open_timeout: close it with 4102, count bothy.terminal.relay.unpaired with bothy.terminal.side = client. b. A host data connection whose identify was accepted, for a session in opening, with no device stream on this node within terminal.open_timeout: close it with 4102 (new on hostData, design note 15), count it with host. c. A session still in opening then ends open_failed with close_detail = open_timeout through M2-44's end function (audit terminal.terminate, terminal_session.changed, which the event-log tail brings to every node). When both sides time out on two nodes, the second end is a compare-and-swap that changes nothing: one audit entry, one event. A device that attaches to a detached session and finds no host data connection on its node gets 4102, and the session stays detached (13: "attach later"). d. A host data connection of a detached session waits for an attaching device until the session ends (terminal.detach_grace); it is not unpaired. A connection that was paired once never counts as unpaired. e. The timer starts when the side's identify is accepted and fires at LoneDeadline, the acceptance time plus terminal.open_timeout, inclusive. Frames of a lone side never leave the node: not into the database, the event log, a signal or another node. No readiness check is involved; /readyz stays 200.

  9. Reconciliation after a listener reconnect. After LISTEN succeeded again (never before, so nothing committed in between is missed), the node reads with HostStatesByIDs the hosts of all its control and data connections, then closes what ended while it was deaf: hosts no longer active with the codes of design note 3; it sends terminal.host_present for each of its eligible control connections at once. Sessions that ended meanwhile need nothing here: the event-log tail has its own LISTEN and poll and reads on from local_head (02 11.2), so M2-44's observer closes their connections whether this listener is up or not. Part of D-M2-47-5.

  10. Locks, order and idempotency. Signals take no row lock; a staged NotifyEphemeral adds the commit-time NOTIFY lock of 02 11.7 only with multi (03 6.6 accepts that cost). Every end of a session goes through M2-44's compare-and-swap; the loser writes nothing. Closing an already closed connection, a second session.close for an ended session and a terminal.host_present for an unknown host are no-ops. Notifications of one sender arrive in commit order, those of several senders in any order; the handler's database reads make the order irrelevant.

  11. Example edge (infra/compose/edge/nginx.conf; Ruling for docs/DECISIONS.md, D-M2-47-9). nginx (BSD-2-Clause) in the image nginxinc/nginx-unprivileged, which runs without root; pin the newest stable release line available at build time as the full tag <version>-alpine in BOTHY_NGINX_VERSION and record version, digest and license in the report. The file is a complete nginx.conf with:

    • pid and all temp paths under /tmp (read-only root file system), listen 8080, access_log off and error_log /dev/stderr crit, because access and error lines carry request lines and client addresses (16 4.4 point 3, SEC-LOG-3, SEC-LOG-5); client_max_body_size 0, because the server enforces its own named limits.
    • map $uri $bothy_terminal_session with the single regular expression ^/api/v1/(?:terminal|hosts)/sessions/(?<sid>[^/]+)/stream$ to the captured session ID, and map $http_upgrade $connection_upgrade (upgrade, empty to close).
    • upstream bothy_terminal_stream with hash $bothy_terminal_session consistent; and the servers api:8080 and api-2:8080; upstream bothy_api with the same servers and the default balancing, for REST, /api/v1/gateway and /api/v1/hosts/connect (24 9 point 1: the control connection may land anywhere).
    • One location for the stream regular expression to bothy_terminal_stream with proxy_buffering off and proxy_request_buffering off, and location / to bothy_api; both with proxy_http_version 1.1, Upgrade $http_upgrade, Connection $connection_upgrade, Host $http_host and X-Forwarded-For $proxy_add_x_forwarded_for (the outer edge's X-Forwarded-Proto passes unchanged).
    • Timeouts (16 4.4 point 1, refined in design note 15): on the stream location proxy_read_timeout and proxy_send_timeout equal to the defaults of terminal.idle_timeout plus terminal.idle_grace, the longest silence of a session the server keeps (24 6.3); on location / equal to the defaults of terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit, which is also above sync.heartbeat_interval of the gateway. A comment above each value names its keys; TestEdgeExample_Config keeps the numbers equal to the defaults in docs/operations/configuration.md, the only file besides the configuration that holds them.

    Reason for nginx: consistent hashing over a value taken from the path is a single documented directive there; the real image in the test proves the routing.

  12. Compose overlay (infra/compose/compose.multi-node.yaml; Ruling for docs/DECISIONS.md, D-M2-47-10). Used as docker compose -f compose.yaml -f compose.multi-node.yaml with the profile direct only.

    • api: ports: !reset [] and BOTHY_CLUSTER__MODE: multi.
    • api-2: extends of api from compose.yaml (same image, command, hardening, volumes with the shared signing-keys key store of ADR 0066, healthcheck, stop_grace_period), ports: !reset [], BOTHY_CLUSTER__MODE: multi, and depends_on written out (as api), because extends need not carry it.
    • worker: BOTHY_CLUSTER__MODE: multi.
    • edge: image nginxinc/nginx-unprivileged:${BOTHY_NGINX_VERSION:-<tag>}, profile direct, read_only: true, cap_drop: [ALL], security_opt: ["no-new-privileges:true"], tmpfs: [/tmp], the local log driver with the options of compose.yaml, the file ./edge/nginx.conf read-only, the port 127.0.0.1:${BOTHY_HTTP_PUBLISH_PORT:-8090}:8080, only the network default, depends_on both api nodes with service_healthy.
    • Header comment: what the overlay is for, that the operator's TLS edge stays in front, that BOTHY_TRUSTED_PROXIES then names the subnet of the network <project>_default (field Subnet of docker network inspect), because the server sees the edge's address and the edge appends the gateway's (16 4.3), and that the profile tailscale runs one api node.
    • Add BOTHY_NGINX_VERSION to .env.example and to section 9 of docs/operations/configuration.md; TestComposeMultiNodeOverlay keeps both equal to the default in the overlay.

  13. Test harness for two nodes (test/terminal/multinode). One pgtest database; nodes A and B assembled in-process through internal/wire in role api with cluster.mode = multi, each with its own public and internal listener on 127.0.0.1:0, the same key store directory from mktemp -d (secrets.signing_key_dir, mode 0700), the same clock.Fake for every Now read; an instance C in role worker for the worker case. The test edge is an in-process httputil.ReverseProxy on 127.0.0.1:0 that passes WebSocket upgrades, with the modes hash (both stream paths by rendezvous hashing over the {session_id} segment, all other paths to a node the test chooses) and split (device streams to A, host data to B, all other paths to a chosen node); the test can switch modes between steps. World: Robin is the owner; Alex is the assignee of the host lab-1 (active, profile shell) with a grant terminal and the session user alex-shell, which the fake host reports; the fake host of M2-44 connects its control connection and its data connections through the test edge with the edge's base URL; the device is the client of M2-44's relay tests. If the merged wiring has one global meter provider, the counter assertions use the process total by side. Timers (the open timeout of the relay, the probe, the end of a probation, the listener's retry) run on the timer source of M2-44's relay: if it follows clock.Fake, the tests advance the fake clock; otherwise the test configuration sets short values of terminal.open_timeout, terminal.host_duplicate_probe_timeout and database.listen_reconnect_backoff_cap, and the tests wait with a bound of a small multiple of them. Exact boundaries are proven by the pure functions of the unit tests, never by waiting. Advance the fake clock between two accepts whose order a test needs.

  14. Telemetry and logs (Ruling for docs/DECISIONS.md, D-M2-47-11). This ticket registers no instrument and no attribute: M2-44 registers and records bothy.terminal.relay.unpaired (D-W-105; signals.md 4.3a, unit {connection}) together with its attribute bothy.terminal.side (client, host; signals.md 5.3, collector allowlist, otel/attr), and M2-46 the other three bothy.terminal.* instruments; this ticket asserts the counter with two nodes. Log records (constant English messages, attributes only from signals.md 5.3): terminal relay side unpaired (warn; bothy.terminal.side, bothy.close.code), terminal signal dropped (warn; bothy.reason malformed), terminal ephemeral listener lost (warn; error.type), terminal ephemeral listener restored (info), terminal duplicate control connection (warn; bothy.close.code). No session, host, person or device ID and no address in any of them.

  15. Specification and documentation text (English, like the files, in the same change):

    • 24 9 "Multiple nodes": the kinds of design note 3 and their receivers, and that the end of a session reaches every node through the log tail without a signal; the presence view and its window (design note 6); the rules of the lone side (design note 8), stating that an opening session ends open_failed with close_detail = open_timeout and a detached one stays detached; the listener, its reconnect and the reconciliation (design notes 5 and 9); what is lost while a node's LISTEN connection is down.
    • 24 3.4: the order of two connections across nodes, rules b and c of design note 7, the probation and the clock assumption. 24 6.1 point 6: reference to the phase rule of 9. 24 13: 4102 with the channels "stream, data connection" and the phase rule. 24 16: row R-TM-31 "Signals between api nodes through NOTIFY ephemeral only with kind and IDs, also for state transitions, probe and presence; the end of a session reaches every node through the log tail; one path for one and for several nodes" with its reason.
    • 16 4.4 "Edge and proxy": point 1 states the idle timeouts of design note 11 (stream paths at least terminal.idle_timeout plus terminal.idle_grace, control connection at least terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit), point 2 names the example (infra/compose/compose.multi-node.yaml, infra/compose/edge/nginx.conf), a new point 4 names the overlay's conditions (profile direct, BOTHY_TRUSTED_PROXIES with the subnet, shared key store).
    • 02 11.6: after its table a paragraph "Terminal signals" (no rows in the table, because they are no event types of 05): the kinds of design note 3, one node local after the commit, several nodes NOTIFY ephemeral in the transaction with host, session or connection ID; that the end of a session goes through the log tail (11.2), not through these signals; and that the module terminal holds its own LISTEN ephemeral connection until M7-17 shares one per process. 02 14.4: a row "LISTEN connection of the terminal" (soft; reconnect, then reconciliation). 02 16 "Multiple nodes in detail": a bullet "Terminal: 24, section 9".
    • docs/api/asyncapi.yaml: hostData.x-close-codes gains 4102 terminal_host_unavailable with client_action "no counterpart on this node within terminal.open_timeout; do not reopen this data connection, wait for the next session.open"; the client_action of 4102 on terminalStream says that an opening session ended open_failed (open again) and a detached one stays detached (attach again later).
    • Threat model: status head "as of milestone M2" if it does not say so yet; 5.16 two rows ("Relay | D | Edge without hash routing splits the stream paths of a session across two nodes | lone side closes after terminal.open_timeout with 4102, metric, no broker, example edge (16 4.4) | T-TM-15" and "Relay | E | End, revocation or second control connection on another node goes unnoticed | end and revocation through the log tail of every node, state transition as a signal in the transaction, reconciliation after reconnect, probe across nodes (24 3.4, 9) | T-TM-05, T-TM-11"); in 6.7 the test columns of T-TM-05 and T-TM-11 name the tests of this ticket, and a new row T-TM-15 "Misrouting and lost signals across several nodes" (L low, I medium; Mitigations as above; Tests TM-T15 and the cross-node tests; Residual risk low: while a node's LISTEN ephemeral connection is down, its control connections of a host disabled or re-paired elsewhere stay open until the reconciliation after the reconnect (the sessions still end through the log tail), and opens through its control connections fail visibly with open_failed).
    • ADR 0068 "Confirmation": a bullet "Two api nodes with and without hash routing (TM-T15)"; the decision stays unchanged; make adr-index.
    • docs/operations/install-compose.md new 5.3 "Several api nodes (overlay)"; infra/compose/README.md (files table, overlay, profile direct only, checks of this ticket); internal/terminal/README.md (mode, signals, listener, presence, probation, lone side, limits, the edge requirement).
    • 03, 04 and 05 stay unchanged: 03 6.6 already allows host and terminal session IDs on the channel. If the build finds that the core contract contradicts a rule here, stop with Verdict: BLOCKED.

  16. Rulings for docs/DECISIONS.md section 10, one row each with date, decision, reason and place: D-M2-47-1 scope (note 1); D-M2-47-2 mode and roles (note 2); D-M2-47-3 kinds, payload and strict decoding, and the expectation of terminal.session_open on every node (note 3); D-M2-47-4 the port with one handler for both modes (note 4); D-M2-47-5 listener, reconnect, reconciliation, soft dependency (notes 5 and 9); D-M2-47-6 presence view and its window (note 6); D-M2-47-7 probe order and probation (note 7); D-M2-47-8 rules of the lone side and 4102 on hostData (note 8); D-M2-47-9 the example edge and its timeouts (note 11); D-M2-47-10 the overlay (note 12); D-M2-47-11 log records, with the counter and its attribute left to M2-44 (note 14).

Tests (write them first)

Unit:

  • internal/terminal/domain: TestSignalCodec (each kind encodes and decodes to the same value; a payload with an unknown field, a missing ID, an extra ID for the kind, a non-UUID, null or an unknown terminal. kind fails; typing.started and other foreign kinds give ours = false without error); FuzzDecodeSignal (any input gives an error, a foreign kind or a signal that encodes back to an equivalent payload; never a panic); TestSignalPayloadBound (the longest signal stays below the protocol constant for the NOTIFY payload); TestOlderConnection (UUIDv7 IDs of two instants in both orders, equal IDs are not older); TestPresenceView (an entry counts at exactly terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit after its last Seen and not one tick later; Gone of one conn_id keeps the other; an unknown host is not connected); TestProbationEnd (twice the probe timeout after the first announcement); TestLoneSideOutcome (device in opening: close 4102, end open_failed with open_timeout, side client; host data in opening: 4102, end, side host; device in detached: 4102, no end, side client; host data in detached: no timer); TestLoneSideDeadline (one millisecond before the acceptance time plus terminal.open_timeout the side is not yet unpaired, at that instant it is).
  • internal/terminal/app: TestLocalSignals (Stage and Send do nothing, Committed calls the handler once); TestUseCasesStageSignals with a recording fake Signals: open stages terminal.session_open; setHostState to disabled and retired stage terminal.host_close; a re-pairing stages terminal.host_close; close by the person and by an owner, storing session.ended and the ends of setHostState stage no signal (they travel through the event-log tail); a failing unit calls Committed never.
  • internal/terminal/adapters/wsrelay: TestHandleSignal_SessionOpenExpectsOnEveryNode with a fake ControlLinks and fake Timers: on a node without a control connection of the host, terminal.session_open registers the expectation through RelayControl.Expect and sends no session.open; a host data connection of the session then pairs there instead of getting 1000; an older device side of the session on that node closes with 4103 and an older unpaired host side with 1000; a node with an eligible control connection registers the expectation and sends exactly one session.open; a closed session or a session of another host registers nothing; the deadline of an expectation without sides ends a session still in opening and nothing for a session that is active in the database; on a host with require_local_confirm = true a side-less expectation that received terminal.session_paired after its grant only removes itself at its deadline (the session paired and waiting for consent on another node stays opening), while without that signal it ends the opening at its deadline; the pairing node sends terminal.session_paired exactly once per pairing of such a session.
  • internal/terminal/adapters/otelmetrics (M2-44's counter): TestRelayUnpairedAttributeSet (the only attribute is bothy.terminal.side with exactly client and host).
  • internal/wire: TestClusterModeKey (enum single and multi, default as 02 22, another value fails the start with a message naming the key); TestTerminalSignalsWiring (single: LocalSignals, no component terminal_ephemeral; multi in api and all: NotifySignals, the component and its stop step in lifecycle.PhaseEventTail; multi in worker: NotifySignals, no listener).

Integration (PostgreSQL 18 via pgtest, the harness of design note 13):

  • TestTMT15_TwoNodesWithHashRouting (TM-T15, first part): edge hash, control connection on B. Open sessions through REST on A until one has its hash node A and one has B (at most a fixed number of opens, closing each after its checks). For each: the fake host gets exactly one session.open on B, valid against host.session.open, with the session's session_id and data_path; device stream and host data connection both arrive on the hash node; handshake message 1, message 2 and later binary frames pass byte for byte in both directions; after the fake host's session.started the row is active; the session token issued on A verifies with the JWKS that B sent in hello; no 4102; bothy.terminal.relay.unpaired unchanged; /readyz of A and B answers 200. Then the device disconnects, attachTerminalSession runs through the other node, the new device arrives on the same hash node and pairs with the waiting host data connection, and a still open older device connection of the session gets 4103 there.
  • TestTMT15_WithoutHashRouting (TM-T15, second part; PLAN row): edge split, open through A, device stream on A, host data on B. Before terminal.open_timeout has passed since each identify, both are open; once it has passed, the device stream is closed with 4102 on A and the host data connection with 4102 on B; bothy.terminal.relay.unpaired rose by one with client on A and by one with host on B; the row is closed with open_failed and close_detail = open_timeout; exactly one terminal.terminate audit entry and one terminal_session.changed for the end; no byte of the device reached the fake host and none of the fake host reached the device; /readyz of A and B answers 200 before, during and after; two log records terminal relay side unpaired with side and code and without any ID or token.
  • TestTMT15_WithoutHashRouting_Attach: a session runs with hash; the edge switches to split; the device disconnects and attaches; the new device lands on the wrong node and gets 4102 after terminal.open_timeout, the counter rises with client, the session stays detached and the host data connection on the hash node stays open; with hash again the next attach pairs.
  • TestCrossNode_SessionOpenOnlyAfterCommit: control on B; a test wrapper around NotifySignals stages terminal.session_open and then fails the unit; the open answers an error, no row exists, the fake host gets no session.open and a test connection with its own LISTEN ephemeral receives nothing.
  • TestCrossNode_SignalsCarryOnlyIDs: a test connection with LISTEN ephemeral records every payload during the TM-T15 tests, the close tests and the host tests; each decodes strictly into its kind; no payload contains the session token, its jti, the profile name, the host name, the session user name, a person ID or a device ID (canary search).
  • TestCrossNode_ForgedSignalOpensNothing: the test sends pg_notify('ephemeral', ...) with terminal.session_open for a random session ID, for a closed session and for a session of another host, and malformed payloads; the fake host gets no session.open; each malformed payload gives one terminal signal dropped; a payload of kind typing.started gives nothing.
  • TestCrossNode_SingleModeSendsNoNotify: one node with cluster.mode = single; open, attach and close work as M2-44's tests expect; the test listener receives nothing and pg_stat_activity shows no backend with application name bothy-terminal-ephemeral.
  • TestCrossNode_CloseReachesRelayAndControl, control on B, relay on A, subtests: person_close (closeTerminalSession through B: device stream and host data closed with 4100 and client_closed on A, the fake host gets session.close with client_closed on B); owner_close (Robin through A: closed_by_owner the same way); host_reported_end (the fake host sends session.ended with process_exited to B: A closes both with 4100); worker_end (an end committed in a unit of instance C in role worker, through the end function the ends of M2-46 and terminal.sweep use: A closes both, B sends session.close). Each subtest compares codes, reasons and frames with the same scenario on one node.
  • TestCrossNode_HostTransitionsCloseEverywhere: control on B, one session relayed on A; setHostState to disabled through A, then a second host to retired, then a re-pairing with issueHostPairingCode: the control and data connections close with the codes of M2-43's single-node path for that state (expected 4111, 4111, 4110) and the device streams with 4100 and the sessions' close_reason, on whichever node holds them; parity with a run on one node.
  • TestCrossNode_DuplicateControlConnection (T-TM-05), subtests: older_answers (fake host H1 on A answers pings; H2 with the same keys connects through B: within terminal.host_duplicate_probe_timeout both are closed with 4112, the host is disabled with duplicate_connection, its sessions end host_disabled, one audit host.connect with code duplicate_connection, host.updated to the owners); older_silent (H1 stops answering pings: after terminal.host_duplicate_probe_timeout H1 is closed with 4009 on A, H2 stays on B, A sends terminal.host_gone); probation_defers_open (an open through A right after H2 connected: session.open reaches H2 only at the end of its probation; with H1 still answering, H2 never gets one); announcement_lost (A's listener is down while H2 connects; after A's reconnect and the next heartbeat of H1 the probe runs and the host ends disabled).
  • TestCrossNode_HostUnavailableParity: no control connection anywhere: openTerminalSession through A and through B both give 409 host_unavailable; control on B: an open through A gives 201; the fake host closes its control connection: an open through A gives 409 host_unavailable; a harness switch drops node B's connections and listener without its stop steps, like a crash, so no terminal.host_gone is sent, and the fake host does not reconnect: an open through A gives 201 until terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit after the last terminal.host_present, then 409 host_unavailable; a session opened in that window ends open_failed with open_timeout.
  • TestCrossNode_ListenerLossReconciles (T-TM-11): control on A, a session relayed on A; the test ends A's listener backend (application name bothy-terminal-ephemeral) with pg_terminate_backend and disables the host through B with setHostState; A's device stream and host data connection still close with 4100 and host_disabled through the event-log tail, while A's control connection stays open and /readyz of A answers 200; after the reconnect A closes the control connection with 4111; one terminal ephemeral listener lost and one terminal ephemeral listener restored.
  • TestCrossNode_StepUpConsumedOnce (ID-T28 across nodes): Alex confirms one fresh step-up; two concurrent openTerminalSession with it, one through A and one through B, give exactly one 201 and one 401 step_up_required, and one row.

Edge and overlay (Docker, test/edge):

  • TestEdgeExample_HashRoutesBothStreamPaths: on a test network, two responder containers of the pinned nginx image with a test config that answers every request with 200 and a header naming the node, with the aliases api and api-2; the edge container with infra/compose/edge/nginx.conf mounted read-only, port 8080 published to 127.0.0.1 on a random port. For many UUIDv7 session IDs, /api/v1/terminal/sessions/{id}/stream and /api/v1/hosts/sessions/{id}/stream reach the same node, and both nodes occur; /api/v1/hosts/connect and /api/v1/gateway are not pinned to one node.
  • TestEdgeExample_Config: in the edge container nginx -t passes and nginx -T shows proxy_http_version 1.1, the upgrade headers, proxy_buffering off on the stream location, hash $bothy_terminal_session consistent only in bothy_terminal_stream, access_log off, error_log at crit, client_max_body_size 0, X-Forwarded-For $proxy_add_x_forwarded_for, listen 8080 and pid under /tmp; the stream timeouts equal terminal.idle_timeout plus terminal.idle_grace and the other timeouts equal terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit, both read from the defaults in docs/operations/configuration.md, and the latter is above sync.heartbeat_interval; the container runs read-only with all capabilities dropped and id -u is not 0.
  • TestComposeMultiNodeOverlay: docker compose --project-directory infra/compose -f infra/compose/compose.yaml -f infra/compose/compose.multi-node.yaml --env-file infra/compose/.env.example --profile direct config --format json. api and api-2 have the same image, command, volumes (with signing-keys), read_only, cap_drop, security_opt, tmpfs, healthcheck, stop_grace_period, depends_on and networks, both BOTHY_CLUSTER__MODE=multi and no ports; worker has BOTHY_CLUSTER__MODE=multi; edge uses the tag of BOTHY_NGINX_VERSION in .env.example, which equals the default in the overlay, publishes only 127.0.0.1:<BOTHY_HTTP_PUBLISH_PORT>:8080, is hardened as in design note 12, is not on internal, mounts the config read-only and waits for both api nodes healthy; every published port of every service binds 127.0.0.1. Case tailscale (profile tailscale only): neither edge nor api-2 exists.

Contract: frames sent on control connections validate against host.session.open and host.session.close of asyncapi.yaml; responses of the operations used validate against openapi.yaml (G8) where the merged tree has the response validator.

Red team focus of the milestone ("misrouting across several nodes", T-TM-05, T-TM-11) is covered by TestTMT15_WithoutHashRouting, TestCrossNode_DuplicateControlConnection, TestCrossNode_CloseReachesRelayAndControl, TestCrossNode_ListenerLossReconciles and TestCrossNode_ForgedSignalOpensNothing; name them in the red team request.

Run FuzzDecodeSignal once locally with go test -run=^$ -fuzz=FuzzDecodeSignal -fuzztime=<duration of your choice> and report duration and result; commit any crashing input as a seed after the fix.

Commands that must pass, from the repository root:

cd services/server && go test -race -count=1 ./internal/terminal/... ./internal/wire/... ./internal/platform/... ./internal/kernel/... ./test/terminal/... ./test/edge/...
make codegen-check
make arch
make register-check
make api-lint
make api-checks
make api-tables
make config-reference
make config-doc-check
make adr-index
make lint
make docs
make check

After committing, git diff --exit-code docs/operations/configuration.md docs/adr/README.md shows nothing after make config-reference and make adr-index.

Acceptance criteria

  • TM-T15 with hash routing: two api nodes with cluster.mode = multi; session.open reaches the node of the control connection over NOTIFY ephemeral only after the commit; both stream paths of a session meet on one node; every node registers the expectation of terminal.session_open, so the session runs whether the control node and the relay node are the same or not; attach works across nodes and replaces an older device side with 4103 on the hash node.
  • TM-T15 without hash routing: each lone side closes with 4102 after terminal.open_timeout, bothy.terminal.relay.unpaired rises with client and host, /readyz stays ready on both nodes, the session ends open_failed exactly once, and no byte crosses nodes.
  • With cluster.mode = single no NOTIFY is sent and no listener runs; M2-43's and M2-44's behaviour is unchanged.
  • Signals carry only a kind and IDs; rolled-back, forged and malformed signals open nothing.
  • An end of a session or a host transition decided on any node, also in a worker, closes the connections on every node with the codes of one node.
  • A second control connection on another node is probed: 4112 and disabled with duplicate_connection when the older answers, 4009 for a silent older one; a new connection gets no session.open during its probation.
  • 409 host_unavailable answers the same on every node; a dead node's entry expires after terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit.
  • A lost listener reconnects, reconciles and never affects /readyz; one fresh step-up opens exactly one session across nodes.
  • The example edge routes both stream paths of a session to one node with the real nginx image, its timeouts and logs follow 16 4.4, and the overlay renders two equal api nodes behind it with ports only on 127.0.0.1.
  • 24, 16, 02, asyncapi.yaml, threat model, ADR 0068, install-compose.md, configuration.md section 9, the Compose README, the terminal README, DECISIONS and both CHANGELOGs are updated.
  • make check is green.

Definition of done

  • Code and tests as specified; doc comment on every public symbol (purpose, parameters, return value, errors, example where useful); comments explain why.
  • README of terminal updated (cluster mode, signal kinds, listener, presence view, probation, lone side, edge requirement, what M7-17 adds); infra/compose/README.md updated.
  • Rulings of design note 16 recorded in docs/DECISIONS.md section 10 (one row each with reason); the changes to 24, 16, 02, asyncapi.yaml, the threat model and ADR 0068 made in the same change.
  • services/server/CHANGELOG.md and the root CHANGELOG.md entries under 'Unreleased'.
  • An ADR only if you take an architecture decision beyond this brief (MADR 4 as in docs/adr/0000-template-madr-4.md, headings and text in English, next free number, make adr-index).
  • make check green.
  • An independent code review over the diff, then the red team gate (docs/quality/red-team-gate.md, verdict JA or NEIN); on NEIN fix with a test per finding and have the delta checked again. Record both verdicts.
  • Land the change as AGENTS.md ("How work lands on main") describes: a pull request from this branch, squash-merged only when review, red team and make check, ci-ok and the review threads are green; then git worktree remove.

Report

Write to {{REPORT_PATH}}:

  1. First line exactly Verdict: DONE or Verdict: BLOCKED <reason>.
  2. Commits (SHA and subject), changed files with line counts, test results with counts (the red run before the change and the green run after), the summary of make check.
  3. Review and red team verdicts and the findings fixed.
  4. What is not proven (for example the timing margins of the probation on loaded nodes, other edges than the example, the overlay started with real images, which M2-48 and M7-17 cover).
  5. Follow-ups that could not be done in this scope, each with its reason, the version, digest and license of the nginx image, every difference between the names in this brief and the merged code of M1 and M2-38 to M2-44 (D-W-100), and spec gaps found.

Then write {{DONE_PATH}} with exit=0 sha=<sha> or exit=1 <reason>.