| D-M7-17-1 |
2026-10-23 |
Payload of the gateway signals on the channel ephemeral: one compact JSON object per NOTIFY with kind (prefix gateway.: gateway.typing with thread and principal, gateway.run with run, thread and version, gateway.agent_presence with agent, state and optionally billing mode and actor, gateway.device with device, gateway session and bind ID, gateway.consumer with agent and gateway session, gateway.hint with recipient, nid, hint_kind, quiet, count and optionally thread, message and target), node and exactly the members of its kind; IDs as canonical UUIDs, codes as lower-case codes of the 05 schemas, never a text. Decoding is strict (unknown, missing or duplicate members, a wrong form or an unknown gateway. kind drop the payload with one log record; a kind outside the prefix is ignored). A node ignores the signals carrying its own node. Place: internal/gateway/domain/cluster.go, 02 11.6. |
The brief's form <kind>:<id>[:<version>] cannot carry the members of a hint or of agent presence without a second format; JSON with a kind prefix is the form of the terminal signals of M2-47 on the same channel (D-M2-47-3), so both routes share one connection and ignore each other's kinds. Codes and numbers are no content in the sense of 03 6.6. |
| D-M7-17-2 |
2026-10-23 |
One LISTEN ephemeral connection per process: platform/pg.EphemeralListener (component ephemeral_listener, application name bothy-ephemeral, stop step in PhaseEventTail) hands every payload in order to the routes of the gateway and of the terminal, and after a lost connection lets each route reconcile once LISTEN succeeded again. It replaces the terminal's own listener of M2-47; the terminal keeps its decoding and reconciliation as terminal/adapters/postgres.SignalRoute. Place: internal/platform/pg/ephemeral.go, internal/wire/gateway_cluster.go. |
02 11.6 announced one shared connection per process from M7-17 on; two connections per process would double the listening sessions for the same channel. |
| D-M7-17-3 |
2026-10-23 |
Device replacement across nodes: every binding of a device's session to a connection (identify or resume of a person) gets a fresh UUIDv7 bind ID and sends gateway.device. A node that holds a connection of that device under another gateway session with an older bind ID closes it with 4017; a signal with an older bind ID than the local one changes nothing (the other node then closes its own on this node's signal). Place: internal/gateway/app/sessions.go (takenElsewhere), gateway.go (DeviceTaken). |
04 6.1 allows one connection per device; UUIDv7 orders the bindings across processes without a clock comparison of the receiver, and two bindings in the same millisecond still have one order, so exactly one of two crossing signals closes a connection. |
| D-M7-17-4 |
2026-10-23 |
Consumer fencing across nodes: a takeover (identify of an agent consumer, or a new REST poll window, which is a new holder) sends gateway.consumer; every other node saves the held acknowledgement of each of its connections of that agent that is not the new holder at once with the guarded statement of agents (Save), and closes it with 4015 when no row is held by its session; without the signal the next regular save does the same. Run starts stay compare-and-swap (04 15.3). Place: internal/gateway/app/consumers.go (Fence, announce). |
02 16 "Agent consumers": the signal only speeds up the fencing of holder_session_id; a lost signal leaves the old connection at most until its next save, and the event IDs and the start CAS prevent duplicate work meanwhile. |
| D-M7-17-5 |
2026-10-23 |
Hints and the other ephemeral events across nodes: the publisher decorator app.Cluster.Publisher delivers locally first and then sends a signal for typing.started, run.updated, agent.presence_changed and notification.created; the receiver rebuilds the event (typing with the place of the thread and expires_at = now plus sync.typing_ttl, run updates into its coalescer, which sends the run document as snapshot, hints with the recipient as audience and the place of the thread), and its hub admits each connection again. command.autocomplete_requested stays on its node: it carries an invocation token and its answer must reach the waiting request of the same process; without a connection of the agent on that node the request ends at once with an empty result (R-AG-14). Lost signals stay lost (at most once, 04 9 point 4). Place: internal/gateway/app/cluster.go, 02 11.6. |
02 11.6 lists autocomplete among the events that take the path with IDs, but neither 03 nor the brief gives a place to store the request with its token, and a token never goes through NOTIFY; the fail-safe answer exists since M2-17. Listed as a spec gap in the report. |
| D-M7-17-6 |
2026-10-23 |
Presence writes and TTL (Design note 2): app.PresenceWriter flushes every cluster.presence_flush_interval in one transaction: the upsert of one row per counted live connection of the hub (gateway session, person, device, focus, idle) with updated_at = now, the delete of the node's rows whose connection ended and the delete of every node's rows older than cluster.presence_ttl; its stop step after the connections closed deletes all rows of the node. A failed write is logged (gateway presence not written, retrying at the next interval) and counted (bothy.gateway.presence.write_failures) and never blocks a connection. The reader keeps, per person, the newest fresh row of each device and ignores rows older than cluster.presence_ttl. A configuration check requires cluster.presence_ttl above twice cluster.presence_flush_interval; defaults 2s and 10s. Place: internal/gateway/app/presencewriter.go, adapters/postgres/presence.go, internal/wire/configkeys_cluster.go. |
Changes of focus or idle reach the worker within one interval without a write per op; one upsert of all rows brings back rows lost with the UNLOGGED table after a database crash; a node that crashed stops refreshing, so its rows age out without a coordinator. |
| D-M7-17-7 |
2026-10-23 |
Wiring by role and mode (Design note 3), wire.GatewayPlan: role all with single runs none of it (in-memory view, no table writes, no NOTIFY); role all with multi sends signals, listens, writes gateway_presence and reads it for the push decision, whose publisher is the gateway's cluster publisher; role api listens and writes presence in every mode (a separate worker sends hints in every mode) and sends signals only with multi; role worker reads gateway_presence, uses notify's live routes and sends hints as signals, and never listens. The info line of M3-23 for a worker without presence is gone. Place: internal/wire/gateway_cluster.go, module_gateway.go, module_notify.go. |
13 12 point 4 and RK-37: separate roles (level 1) need the hint path as much as several api nodes (level 2); the channel carries only hints there, and role all with one node keeps the cost of NOTIFY at zero (03 6.6). |
| D-M7-17-8 |
2026-10-23 |
One key store of the signing keys for all processes, per target: Compose mounts the volume signing-keys into api, api-2 and worker; a generic VM uses one directory on the host; a target with processes on several hosts needs a shared file system with POSIX permissions or one host for all api and worker processes. A key store per process is not supported. TestSharedKeyStore proves that a key rotated on one node signs and verifies on the other within signing_key.cache_ttl. Place: 16 5.2 and 4.5, docs/operations/multi-node.md. |
ADR 0066 keeps the private keys in a writable directory; a token signed by one node must verify on every node. |
| D-M7-17-9 |
2026-10-23 |
Differences from the brief's names, following the merged code (D-W-100): the table-backed presence.View lives in internal/gateway/adapters/postgres (PresenceView, the owner of the table) and is wired in internal/wire, notify is unchanged; the fencing needed no change in internal/agents or internal/runs (their guarded Save and the start CAS exist since M2-07 and M2-09); TestPresenceTTL is a test of the adapter on pgtest, the other integration tests are in test/gateway/multinode; perf/gateway_fanout_multi runs two server processes (role all and role api) behind an in-process edge and needed harness.StartProcessNode; the retention participant of gateway_presence is not registered because the coverage test of M7-05 is not on the base branch (the flush deletes stale rows itself). Place: this branch. |
Boundaries of the brief: follow the merged names and list each difference. |
| D-M7-17-10 |
2026-10-23 |
Agent presence across nodes is decided from the stored row (the newest heartbeat of every node), never from one node's memory alone, at one place, the presence tracker: (1) a heartbeat that takes the agent over on this node (the entry was not fed here since it was filled, the row holds a newer heartbeat of another node, or the agent was offline or unknown here) is stored at once with the monotone PersistHeartbeat and announces its state again even when the published pair is unchanged; (2) before the deadline timer sets an agent offline it reads the row outside the entry lock; a row with a newer heartbeat is adopted (report, state, last heartbeat) and the deadline moves to that heartbeat plus agent.presence_timeout plus agent.heartbeat_persist_interval, the most the feeding node's row can lag; nothing is published before; a failed read keeps the verdict of this node; (3) Statuses, getAgent and listAgents use the entry only when the row holds no newer heartbeat. Spec 09 notes the immediate store. Found by the red team and the delta red team; TestPresenceAcrossNodes moves an agent between two trackers on one store with a fake clock. Place: internal/agents/app/presence.go (expire, memory, Statuses), heartbeat.go (takeover, persistNow). |
Since M7-17 an offline of one node reaches the clients of every node (gateway.agent_presence); the node the agent left must see the new node's heartbeat before its own deadline, independent of the flush phase, and a node's stale entry must not answer reads; the grace of one persist interval makes the verdict independent of the ratio of persist interval and timeout, so no new configuration check is needed. |