| D-M2-47-1 |
2026-10-20 |
Scope: besides session.open over NOTIFY ephemeral and the hash rule of the edge, M2-47 builds the probe of an older control connection across nodes (24 3.4), the closing of a host's connections on every node at a transition or re-pairing (24 3.1, 3.3) and the check "host connected" across nodes behind 409 host_unavailable (24 6.1 point 2, 13). The end of a session needs no signal: it reaches every api node through the event log tail (M2-44 Design note 11), which M2-47 proves with two nodes. The code stays node-neutral; with cluster.mode = single none of it runs. |
Without these paths cluster.mode = multi would keep the control connection of a disabled or re-paired host open on another node, hide a copied key and answer 409 host_unavailable by chance of the node; no other ticket builds them (M7-17 covers gateway, runs and agents). Place: 24 section 9, internal/terminal. |
| D-M2-47-2 |
2026-10-20 |
Mode and roles: cluster.mode is the enum single or multi of 02 22 (default single, place Server), registered in internal/wire/configkeys_cluster.go with chapter02. single: LocalSignals, no listener, no NOTIFY, no presence view, no probation. multi in the roles api and all: NotifySignals and one EphemeralListener per process. multi in the role worker: NotifySignals without listener. |
Keys enter with their first consumer (M1-07); the worker holds no connection, and the ends it commits need no signal. Place: internal/wire/terminal_cluster.go, 02 22. |
| D-M2-47-3 |
2026-10-20 |
Kinds and payload: terminal.session_open (host_id, session_id), terminal.host_close (host_id), terminal.host_present and terminal.host_gone (host_id, conn_id), terminal.session_paired (host_id, session_id); compact JSON with exactly these members, each a UUID string, never null. Decoding is strict: unknown or missing members, a non-UUID or an unknown kind with the prefix terminal. drop the payload with one log record; a kind outside the prefix is ignored without a log. terminal.session_open registers the expectation on every api node (RelayControl.Expect with the device of the row and the receipt time as grant time); only the node of an eligible control connection sends session.open. On a host with require_local_confirm = true a side-less expectation that received terminal.session_paired only removes itself at its deadline. The receiver always reads the state from the database. |
The node the edge hashes the session to must hold the expectation whichever node served the REST call; late, repeated, reordered or forged signals then change nothing wrong; M7-17 adds its own kinds to the same channel; only the fact of a pairing may spare an opening waiting for local consent, because the server does not know host_service.local_confirm_timeout. Place: domain/cluster.go, 24 section 9, 02 11.6. |
| D-M2-47-4 |
2026-10-20 |
Port app.Signals (Stage in the unit of work, Committed after it, Send outside any unit) with one SignalHandler for both modes: LocalSignals (single) calls the local handler in Committed; NotifySignals (multi) executes NotifyEphemeral on the unit's transaction in Stage and does nothing in Committed, because every node, the sender included, receives the notification through its listener. No use case calls the relay directly after a commit any more. |
PostgreSQL delivers a staged notification only after the commit and never after a rollback; a local shortcut on the sending node would act twice or need a second code path, so the single-node behavior is the multi-node behavior with a local transport. Place: app/signals.go, adapters/postgres/signals.go. |
| D-M2-47-5 |
2026-10-20 |
Listener: EphemeralListener opens its own connection (bothy-terminal-ephemeral), executes LISTEN ephemeral, hands each signal in order to the handler, logs a loss once, retries at once and then every database.listen_reconnect_backoff_cap, and after LISTEN succeeded again runs the reconciliation (host states of all its control and data connections by HostStatesByIDs, close what is no longer active, announce every eligible control connection at once). Soft dependency like the tail's LISTEN: no readiness check; stop step terminal_ephemeral in lifecycle.PhaseEventTail; an unexpected end of its goroutine goes to lifecycle.Runner.Fail. One listener per process. |
Reconciling only after LISTEN succeeded misses nothing committed in between; ends of sessions reach the relay through the log tail, which has its own LISTEN and poll; no new key, because 02 22 already bounds a LISTEN reconnect. Place: adapters/postgres/ephemeral_listener.go, 02 14.4, 24 section 9. |
| D-M2-47-6 |
2026-10-20 |
Presence across nodes: the node of a control connection sends terminal.host_present after its first report and after each heartbeat, and terminal.host_gone when it closes; each node keeps a domain.PresenceView of remote entries, which count while no more than terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit has passed since their last announcement. "Host connected" is an eligible local control connection or a counting remote entry. |
03 has no column for the connection of a host; the window is the silence after which the holder closes the connection itself, so a dead node's entry counts at most that long and an opening it lets pass ends open_failed with open_timeout. Place: domain/cluster.go, adapters/wsrelay/cluster.go, 24 section 9. |
| D-M2-47-7 |
2026-10-20 |
Second control connection across nodes: every accepted control connection gets a conn_id (UUIDv7 on the clock of its node); a node ignores announcements of its own connections; the node of the older connection probes it with M2-43's probe (4009 without answer, else the duplicate verdict in one unit that stages terminal.host_close, so every node closes with the code of the state); the node of a newer connection announces it at once when it hears an older one. Probation: no session.open before twice terminal.host_duplicate_probe_timeout after the first announcement; a due session.open waits and is checked again in the database at the end. The order rests on synchronized clocks. |
Otherwise a host service with copied keys on another node could receive session.open and finish the handshake before the probe decides (T-TM-05); the probation covers the probe window plus notification and commit. Place: 24 3.4, adapters/wsrelay/cluster.go. |
| D-M2-47-8 |
2026-10-20 |
Lone side: a device stream or a host data connection of an opening whose identify was accepted and that finds no counterpart on its node within terminal.open_timeout closes with 4102 (new on hostData) and counts bothy.terminal.relay.unpaired with its side; the opening then ends open_failed with close_detail = open_timeout; a lone device of a detached session gets 4102 and the session stays detached; a host data connection of a detached session waits until the session ends; a side paired once never counts as unpaired. The deadline is inclusive. Frames of a lone side never leave the node; /readyz stays ready. |
A misconfigured edge fails visibly instead of passing bytes through a broker (24 9 point 4, Ruling T6); the same rules hold on one node and on several. Place: domain/cluster.go, 24 6.1 point 6, 9 and 13, asyncapi.yaml. |
| D-M2-47-9 |
2026-10-20 |
Example edge infra/compose/edge/nginx.conf: nginx, pid and temp paths under /tmp, listen 8080, access_log off, error_log /dev/stderr crit, client_max_body_size 0; map of the stream path to the session ID and upstream bothy_terminal_stream with hash $bothy_terminal_session consistent; upstream bothy_api for REST, the gateway and /api/v1/hosts/connect; the stream location without buffering; timeouts on the stream location equal to terminal.idle_timeout plus terminal.idle_grace and on location / equal to terminal.host_heartbeat_interval times terminal.host_heartbeat_miss_limit, each value with a comment naming its keys. |
Consistent hashing over a value taken from the path is one documented directive in nginx, and the real image in the test proves the routing; access and error lines carry request lines and client addresses (SEC-LOG-3, SEC-LOG-5); the edge never cuts a session before the server would end it. Place: 16 4.4 points 1 to 3, test/edge. |
| D-M2-47-10 |
2026-10-20 |
Compose overlay infra/compose/compose.multi-node.yaml, only with the profile direct: api without ports and with BOTHY_CLUSTER__MODE: multi; api-2 extends api (same image, command, hardening, volumes with the shared signing-keys, healthcheck) without ports and with depends_on written out; the worker with multi; edge in nginxinc/nginx-unprivileged (BOTHY_NGINX_VERSION), read-only, without capabilities, no-new-privileges, tmpfs /tmp, published only on 127.0.0.1, waiting for both api nodes healthy. The header names the operator's TLS edge in front and BOTHY_TRUSTED_PROXIES with the subnet of <project>_default. |
tailscale serve has no hash rule, so the profile tailscale runs one api node; the server sees the edge's address and the edge appends the gateway's (16 4.3); the hardening equals the server containers. Place: 16 4.4 point 4, infra/compose/README.md, docs/operations/install-compose.md. |
| D-M2-47-11 |
2026-10-20 |
Telemetry and logs: M2-47 registers no instrument and no attribute; M2-44 registers and records bothy.terminal.relay.unpaired with bothy.terminal.side (client, host), which M2-47 asserts with two nodes. Log records with constant English messages: terminal relay side unpaired (warn; side, close code), terminal signal dropped (warn; bothy.reason malformed), terminal ephemeral listener lost (warn; error.type), terminal ephemeral listener restored (info), terminal duplicate control connection (warn; close code). No session, host, person or device ID and no address. |
Telemetry only as the catalogue allows (signals.md 2 and 4.3a, SEC-TERM-1, SEC-LOG-3, SEC-LOG-5); the counter and its attribute belong to M2-44 (D-W-105). Place: adapters/postgres/ephemeral_listener.go, adapters/wsrelay. |
| D-M2-47-12 |
2026-10-20 |
The node of an eligible control connection sends session.open for every session that is not closed (opening, active or detached) of an active host, not only for opening. |
M2-44 sends one session.open per grant including attach (24 6.2, D-M2-44-4); a host data connection always needs a fresh session.open. Place: app/cluster.go (onSessionOpen), 24 section 9. |
| D-M2-47-13 |
2026-10-20 |
terminal.host_close closes the control connections only; data connections and device streams close with the end of their sessions through the log tail, which every transition out of active commits in the same transaction. The fallback of confirmHost after terminal.fingerprint_max_attempts also stages terminal.host_close. A duplicate verdict on a confirming host puts it back to pairing, so other nodes close with 4110 (the probing node closes its own connections with 4112). |
Parity with one node, which closes them the same way; the fallback closed the connection after its commit on one node. Place: app/cluster.go (onHostClose), app/pairing.go, app/control.go, 24 3.4 and 9. |
| D-M2-47-14 |
2026-10-20 |
The lone side rules change M2-44 in the points 24 9 and 13 decide: a lone host data connection of an opening closes with 4102 (M2-44: 4100 with open_failed); each side has its own timer from its acceptance (M2-44: the slot deadline from the grant); the slot deadline waits while an unpaired side is present; and the end at open_timeout is a compare-and-swap on opening (app.Service.RelayOpeningTimedOut), so two nodes write one end. |
The brief (Boundaries) asks to change the merged relay where it differs from Design note 8 in a point 24 9 and 13 decide. Place: adapters/wsrelay/relay.go, app/cluster.go, 24 6.1 point 6, 9 and 13. |
| D-M2-47-15 |
2026-10-20 |
A slot that never saw session.started reads the session state from the database at its deadline; with several nodes the session.started hook of the control node detaches only a session it paired itself. |
The pairing may live on another node; the deadline of the pairing node reads the state and detaches a pairing that ended before session.started. Place: adapters/wsrelay/relay.go (deadline, Started), 24 section 9. |
| D-M2-47-16 |
2026-10-20 |
Names follow the merged code: domain.Side and its values come from M2-44 (relay.go); the hub of adapters/wsrelay (Hub) is the SignalHandler and hands terminal.session_open and terminal.host_close to app.Service.HandleSignal, which reads the rows and calls the existing ports RelayControl.Expect and ControlLinks.OpenSession; app.New binds LocalSignals over the Service itself, so a process without wiring behaves like one node; the listener is built in the start step terminal_ephemeral (internal/wire/terminal_cluster.go), and its constructor takes the dialer of pg.Pool.DialDedicated. |
The brief lets merged names win over the expected ones (Boundaries, D-W-100). Place: internal/terminal, internal/wire. |
| D-M2-47-17 |
2026-10-20 |
Re-sending ready (setHostLlmCredentials, confirmHost) stays local to the node that served the call; with several nodes a control connection on another node gets the new declaration at its next login. |
The brief defines no signal kind for it. Listed as a gap for M7-17. Place: app/hosts.go, app/pairing.go, internal/terminal/README.md. |
| D-M2-47-18 |
2026-10-20 |
conn_id on the channel ephemeral is an in-memory connection ID (UUIDv7 of the accepting node). |
An ID like those 03 6.6 lists; it names no row and no person. Place: domain/cluster.go, adapters/wsrelay. |
| D-M2-47-19 |
2026-10-20 |
NotifyPayloadMax is 8000 bytes, a protocol constant in domain/cluster.go. |
It is the PostgreSQL limit of a NOTIFY payload in the default build, not a configuration key. Place: domain/cluster.go. |
| D-M2-47-20 |
2026-10-20 |
nginx image nginxinc/nginx-unprivileged:1.30.6-alpine (stable line 1.30, BSD-2-Clause); the example edge adds proxy_request_buffering off and proxy_max_temp_file_size 0 on location /, and the overlay restarts the edge with its api nodes (depends_on with restart: true). |
The edge never buffers bodies in its tmpfs and never keeps stale upstream addresses (nginx resolves the api nodes once at start). Place: infra/compose/edge/nginx.conf, infra/compose/compose.multi-node.yaml, infra/compose/.env.example. |
| D-M2-47-21 |
2026-10-20 |
With cluster.mode = multi a side that reaches the relay builds the expectation of the current grant from the row (app.Service.RelayExpectation), and an expectation carries its grant (the token jti of the row); RelayControl.Expect with the grant the slot already holds changes nothing. A data connection of a session that is not closed therefore never gets 1000 for lack of an expectation on its node. |
The REST answer, session.open on the control node and terminal.session_open on the hash node race: a device or data connection could reach the hash node before its listener delivered the signal and would be refused although the database holds the grant. The receiver reads the state from the database (design note 3), and a signal that arrives later or twice must not reset a pairing. Place: adapters/wsrelay/relay.go (ensureSlot), app/cluster.go. |
| D-M2-47-22 |
2026-10-20 |
With several nodes the relay learns from the log tail that a session started (terminal_session.changed with active or detached): its slot is started, its deadline met, and a pairing that ended before session.started detaches the session. |
session.started arrives only on the node of the control connection; without it the hash node would end a started session at its opening deadline or never detach it when the device leaves. The tail reaches every api process already (02 11.2); no signal kind is added. Place: adapters/wsrelay/relay.go (observeStarted). |
| D-M2-47-23 |
2026-10-20 |
Test seams for two nodes in one process: wire.Deps.TerminalSignals, TerminalListenerGate, TerminalMetrics and TerminalModule, and in test/harness the options PublicURL, Role, OnListen and StartNode (a second process of the same installation behind one edge); the Fake-Host reports the close of its control connection (ControlClose, ControlEnded). The subtest worker_end commits the end in the worker through RelayOpenFailed, the one end path that terminal.sweep and the ends of M2-46 will use, because neither exists yet; the cross-node subtests compare with the codes M2-44's single-node tests assert instead of running each scenario twice. |
Design note 13 needs two nodes on one database and a crashed node; the harness started one server per database. All seams are nil in production. Place: internal/wire/serve.go, test/harness/harness.go, test/terminal/fakehost/fakehost.go, test/terminal/multinode. |
| D-M2-47-24 |
2026-10-20 |
The brief's TestHandleSignal_SessionOpenExpectsOnEveryNode lives in app (the handler of terminal.session_open is app.Service.HandleSignal behind the hub, D-M2-47-16); the relay parts of it (deadline of an expectation without sides, terminal.session_paired once per pairing, the lone sides) are TestRelay_SidelessExpectationDeadline, TestRelay_SessionPairedSentOnce and TestRelay_LoneSides in adapters/wsrelay. The stacked build changes M2-44's relay only in this ticket's own commits, as the brief asks for the points 24 9 and 13 decide (D-M2-47-14, D-M2-47-15). |
Names follow the merged code; one test per layer. Place: app/handle_signal_test.go, adapters/wsrelay/cluster_internal_test.go. |
| D-M2-47-25 |
2026-10-20 |
With several nodes a terminal_session.changed with detached for a session whose slot on this node is started and paired makes the session active again (RelayActivate, which changes only a detached row). |
When the control connection of a host ends on one node, M2-43's ConnectionLost detaches every active session of the host, also one relayed on another node whose device stream and data connection are alive; on one node the relay closes both sides first, across nodes the row would stay detached while the person keeps typing (code review M2-47, Major). Place: adapters/wsrelay/relay.go (observeStarted), test TestCrossNode_ControlLossKeepsRemotePairing. |
| D-M2-47-26 |
2026-10-22 |
The ticket was built on an integration branch and lands on main as a port: one branch from main with only the ticket's own commits, adapted to main, one pull request, squash merge. The chapters and documents on main are English files (02-architecture.md, 16-operations.md, 24-terminal.md, the threat model, ADR 0068); the added text keeps its wording and sits at the same places, chapter references in code and comments name the English files, the key cluster.mode names 02-architecture.md, the changelog entries and these rulings are fragments, and generated files (configuration reference, config schema) are regenerated, not merged. |
Integration branches are retired; every ticket lands on main on its own (as D-M2-44-18). No behavior changes. |
| D-M2-47-27 |
2026-10-22 |
With cluster.mode = multi the signal terminal.session_open is staged in the unit of the opening or attach, so the node of the control connection may send session.open before the recheck after the commit (24 6.1 step 4, D-M2-44-8) has decided. When the recheck fails, the session token is never returned, the recheck ends the session in its own unit, and the end reaches the host through the log tail of every node (session.close). |
The host service starts a PTY only after Noise message 1 carries the single-use session token, so a session.open without a returned token opens nothing; staging in the transaction keeps the signal from being lost between the commit and the recheck. What D-M2-44-8 guarantees with multi is that no token leaves the server after a failed recheck. |