| D-M7-09-1 |
2026-10-25 |
Measurement window (design note 1): a round reads the minute buckets of stats_minutes that end within the last status.check_interval, at least the last complete minute (domain.Window, half-open [from, to) of bucket_start). api.http is 1 - good/count of sli.api_availability, push.apns and push.fcm are 1 - good/count of sli.push_acceptance of their provider (rejected tokens are not counted there, 25 3.2), sync.fanout fails when the mean latency sum/count of sli.delivery_latency is above status.fanout_lag_degraded (it has no outage level). A window without traffic is no evidence and changes nothing; a ratio counts strictly above its threshold. The read goes through the new port stats.SLIWindows, because only stats runs SQL on its table. |
16 12 names "since the last check" and the source of every check; the own measurement is written per minute and per process, so the window is the minutes the last interval covers. No traffic must not turn a quiet instance red. |
| D-M7-09-2 |
2026-10-25 |
Standard components (design note 2): a component has one health_source, so the four standard components carry a group key that names the checks of their part of 16 12: api (api.http, sync.fanout), push (push.apns, push.fcm), files (files.storage, files.scan), search (search.query); the outcome of a group is its most severe check, no evidence counting only when no check has evidence. An owner may also set a single key or agent.presence with agent_id. status.probe creates every standard component whose group key no component carries, under its name or, when that is taken, the name with the key appended; a deleted standard component comes back with the next round (16 12 makes the four parts the minimum). The job runs at the start of a worker as well, so the components exist from setup on. |
The data model has one health_source per component, the spec four parts with two checks each for three of them. The owner keeps names, texts and order; the minimum stays. |
| D-M7-09-3 |
2026-10-25 |
Memory of the job: the state a component had after the last round, since when, since when it is out of operational and when the current run of failing rounds began live in the memory of the worker process that runs status.probe (domain.Track). The memory is valid only while that process saw a round at least every three status.check_interval and the stored state is the one it left; otherwise it starts again from the stored state with status_components.updated_at as the time (a change of the owner moves that column, so a new start can only delay the timers of the state and of the incidents). Whether an outage already has its automatic incident is never decided by this memory, but by stored rows (D-M7-09-7), so a restart, a gap or several workers cannot open a second one or suppress a due one. |
03 has no column for these times and this ticket does not edit the core contract (P-11); the scan health of files.scan is per process as well (D-M3-04-11). A delayed degraded is harmless; a premature one would page the owner. |
| D-M7-09-4 |
2026-10-25 |
Agents: agent.presence reads the stored agents.reported_runtime_state through the new port agents.StatusAgents (every process answers the same, also a worker that tracks no heartbeat): online passes, degraded fails (so degraded after status.degrade_after), offline is no answer (major_outage), unknown (never reported) is no evidence; a stored online or degraded whose last heartbeat is older than agent.presence_timeout plus agent.heartbeat_persist_interval counts as offline, because the node that tracked it stopped without storing the end. Only agents with an active principal get a component (name: the display name, or with the handle appended when taken; description: the handle). agents.AgentDisabled removes the agent's components in the transaction of agent.disable (subscriber registered last); the next round removes them for any agent that is no longer active and removes presence components whose agent was deleted. |
16 12 "State from agents.reported_runtime_state". An agent that was never connected must not open incidents; a crashed node must not leave an agent green for ever. |
| D-M7-09-5 |
2026-10-25 |
Visibility and events: a component of an agent (an agent_id or the check agent.presence) is shown only with agent.read on that agent, never on the public page, and never counts for overall_state; an incident is shown with at least one visible component (with only the visible ones), or with none at all when the owner opened it; an automatic incident whose component is gone (a disabled agent, a deleted component) is shown to nobody, because its title names the component; so is an incident of the owner whose last component was the component of an agent and is gone (status_incidents.hidden, set before every deletion of components), because its text may name the agent and an incident without components is shown to everybody and on the public page. status.updated (receive action status.read, resource instance without an id) names in changed_component_ids and changed_incident_ids only components that belong to no agent and their incidents; a change that touches only agent components (and incidents shown only through them) appends no event at all, also when status.probe creates or removes the component of an agent or agents.AgentDisabled removes it. overall_state of the page and of the event is the worst of the other components in the order operational, maintenance, degraded, partial_outage, major_outage. |
Every reader with status.read receives the event and the overall state, but 16 12 lets people see only agents with agent.read; an id, a name or a state of a foreign agent must not reach them, and neither must the fact that one changed: an event without ids at the moment a foreign agent comes or goes is such a signal (the noise invariance AZ-P5 of test/leak caught it in CI). |
| D-M7-09-6 |
2026-10-25 |
Retention (retention.ops, kernel class job retention.ops, participant adapters/postgres.Retention): incidents resolved before now minus retention.ops go with their updates and component links (cascade); backup runs started before it go, except the latest succeeded run of each kind (ties on started_at both stay). Open incidents never go. status_components carries no end time: it is the live configuration of the page and leaves through the owner, status.probe and agents.AgentDisabled; the coverage test lists it as such. |
20 4.1 class ops; bothy migrate up and the restore test read the latest succeeded run per kind (brief requirement 16). |
| D-M7-09-7 |
2026-10-25 |
Automatic incidents (design note 3): one per outage. An outage begins when the component leaves operational and ends after status.resolve_after of operational; the job then resolves the open automatic incidents of the component and records the end with its own update (state resolved, created_by NULL, the resolved text) also on the newest automatic incident that a person resolved before, whose resolved_at and texts stay. A new incident opens when major_outage has lasted status.incident_after by the memory of the job (never by updated_at, which an owner's change of the name moves) and the newest automatic incident of the component, if any, is resolved and carries the job's end. So neither an owner's edit, a round of degraded in between, a short recovery, a restart, a gap of rounds nor an incident the owner resolved during the outage gives a second one, and a later outage always gets its own. The incident has started_at the start of the outage as the memory knows it, kind incident, impact major, title <component name>: major outage and a first update without measurements. The automation adds only its own updates and never changes an incident of the owner or anyone's text. degraded opens no incident. |
16 12 "Incidents"; the owner's decision to close an incident stands. Review and red team found that updated_at as the start of the outage let an edit, a flap (round 1) or a restart after the owner closed the incident (round 2) open a second one, and that deciding by the memory instead suppressed the incident of a later outage after a restart (round 3); the class fix decides by stored rows only (tests TestProbe_OneIncidentPerOutage, TestProbe_RisingRatioAndRestart, TestIncidentAutomation). |
| D-M7-09-8 |
2026-10-25 |
Owner operations: a taken name, an unknown health_source, agent.presence without agent_id, an agent_id with another check or naming no agent, an unknown after_component_id, an unknown component of an incident, an empty or invalid text, a name longer than 200 characters (the btree entry of the unique index on lower(name) is limited in size) give 400 validation_failed with the field (codes taken, invalid, required, empty, too_long, invalid_enum), because the contract lists no 409 on these operations. sort_key holds fixed-width decimal numbers and a create or a move numbers the whole list again. A create without state is operational, an incident without state investigating (scheduled for a maintenance); resolved sets resolved_at once, another state clears it. An update without effect writes nothing, appends no event and is audited with no_change. maintenance belongs to the owner: the automation leaves such a component alone; partial_outage is only set by the owner. Every write is audited as status.manage with the operation and the id, never a text. |
06 4 items 3 and 4, R-API-27; the page has few components, so renumbering is simpler than a fractional key in a second module. |
| D-M7-09-9 |
2026-10-25 |
Merged names win over the brief (D-W-100): the inbound port is status.Status (already named in 02 4.3), the use cases are app.Page and app.Probe; the check sources are the new ports stats.SLIWindows, files.StorageProbe (probe object status-probe/<process id>, one per process so two workers never delete each other's object), search.Probe (the message search for a fixed term in the zero channel id, under search.statement_timeout, without authorization) and agents.StatusAgents, plus the existing files.ScannerHealth; the class job is retention.OpsJob in kernel/retention, whose JobClasses now includes ops. The checks of a round run at the same time outside the transaction, each bounded by status.check_interval, so River sets no timeout of its own for status.probe. The new cross-check status_error_ratios refuses status.error_ratio_degraded above status.error_ratio_outage. |
The brief names the sources by module; the merged code had no read port for stats_minutes, no storage or search probe and no list of agents with presence. |
| D-M7-09-10 |
2026-10-25 |
Review and red team round 1 (both NEIN), fixed with a test each: a second automatic incident for one outage after an owner's edit or a flap (D-M7-09-7); the check of agent.read per component of getStatus ran outside filter mode and audited a denial for every hidden agent on every read (now authorize.Filter, TestVisibility counts no agent.read denial). Taken from the minors: an incident of the owner that names only agent components is not named in status.updated either. Delta red team (NEIN): after a restart or a gap the seeded memory took updated_at as the start of the outage and reopened an incident the owner had resolved; the following re-review (NEIN): deciding that by the memory suppressed the incident of a later outage after a restart. Both fixed at the class: the job records the end of an outage in the incident rows (D-M7-09-7). |
Blocker and Major rule of the review budget; the other minors are listed in the report. |