Skip to content

M2-26 Outgoing webhooks after Standard Webhooks

No Date Ruling Reason
D-M2-26-0 2026-10-15 The ticket is built on feat/M2-26-webhooks-outgoing from int/m1-wave19 (b380a4e) in its own worktree and is not merged locally; the PR lane ports it. The system action webhook_out.transition and R-AZ-27 (05 section 4.11, D-W-104), which main has and the base lacked, come over as the first commit of the branch (catalogue entry, rule row, generated tables and Go catalogue, the system row of the requirement table). The base still holds the German chapter files; this ticket writes its additions to them in English (owner rule "everything English"), so the PR lane can carry them to the English chapters of main unchanged. Launcher task over the brief's merge step; Design note 1 a names the missing spec state, which the task allows to port.
D-M2-26-1 2026-10-15 The consumer creates delivery rows only for active and paused subscriptions; a disabled subscription gets none, so events of that time are not delivered (webhooks.md 5 point 4). A disabled target must not pile up rows; a re-enable continues with the remaining pending rows and new events.
D-M2-26-2 2026-10-15 The delivery quota pauses on the crossing: when an insert brings an active subscription from below webhook.max_pending_per_subscription to it, the subscription becomes paused (cause max_pending); rows keep being created while paused; the open count is read once per round with a bound and tracked in the memo of the round. No event is lost, and a resumed subscription with a large backlog is not paused again before it drained and filled up again.
D-M2-26-3 2026-10-15 One job kind apps.webhook_deliver with only the subscription as argument serves every trigger (round, chain, retry, lease wake-up, owner resume, redelivery, app.resume); an attempt claims the due head under the subscription row lock, takes a lease of webhook.delivery_timeout with a wake-up job, sends outside any transaction and records in a second unit of work, discarding the result when the head is no longer pending with the claimed attempt. Strict order per subscription with at most one attempt in flight; a crash after the send leads to a second send with the same webhook-id (at least once).
D-M2-26-4 2026-10-15 webhook_subscriptions.signing_key_kid names the active key that signed the last accepted delivery; at creation the active key. 03 requires the column; it then names the key a receiver needs most.
D-M2-26-5 2026-10-15 A gap is detected at the head (its event is no longer in the log: every pending row below the smallest retained seq becomes dead with event_expired) and from the lost range of a consumer round (sync.ConsumerLossHandler: every live app subscription becomes disabled, all its pending rows dead); both disable with cause gap and name the lost range in the audit entry. Among existing rows the head rule finds exactly the due deliveries; the loss of the consumer itself produces no rows, so only the round can see it. A re-enable never loops.
D-M2-26-6 2026-10-15 Statuses pending, succeeded, failed (after 410, terminal), dead (after webhook.retry_max_age, not_authorized, not_subscribed, event_expired); last_error from the closed list timeout, connection_failed, tls_failed, address_blocked, host_not_allowed, http_status, throttled, gone, not_authorized, not_subscribed, event_expired; throttles (429, 502, 503, 504) are retried but not counted; every other status, 3xx, timeouts and refused or failed connections count. webhooks.md 4 separates throttle and failure; the response body is never stored (03).
D-M2-26-7 2026-10-15 Backoff after the n-th failed or throttled attempt: uniform in [base, min(cap, base * 2^(n-1))] with an injected random source; for a throttle with a valid Retry-After the larger of backoff and Retry-After; no retry after created_at + webhook.retry_max_age (cut to it), and a failure at or after it makes the delivery dead and the subscription disabled; the deadline is checked only after a failed attempt. Deliveries take created_at from the clock of the service. Exponential with full jitter between base and cap (webhooks.md 4 point 1); a head that waited in a paused subscription still gets its attempt.
D-M2-26-8 2026-10-15 Entries of webhook.allowed_hosts have the form host[:port] (without a port only 443); an address must parse, use https, have no user info, fragment or opaque part, and its canonical host and port must match an entry (domain.ValidateTarget): host or port outside the list is 400 webhook_host_not_allowed, every other defect 400 validation_failed at /url. The same rule checks manifest addresses (replacing the host-without-port comparison of D-M2-24-5) and is the host policy of the webhook egress client. One rule for manifests, subscriptions and the egress client; a list changed after a restart blocks old subscriptions at delivery (host_not_allowed, counted).
D-M2-26-9 2026-10-15 The types an app subscription may name are the keys under webhooks: of openapi.yaml minus the types with receivers: agent in actions.yaml, as a hand-written table in domain held equal to both files by TestAppSubscribableMatchesContract. webhooks.md 7 lists the agent-only rows; a test catches a drift of either file.
D-M2-26-10 2026-10-15 Signing set: v1a with every signer of instance.Keys.Signers(webhook_out) (active first, then retiring keys whose not_after lies ahead); v1 with the key of hmac_generation and, inside webhook.secret_rotation_overlap, the previous one. Header entries separated by one space, v1a first in signer order, then v1 newest generation first, standard base64 with padding. HMAC key = HMAC-SHA256(master, "webhook-v1:" + canonical subscription id + ":" + decimal generation), the master key the hex-decoded value of webhook.hmac_master_key_ref; the secret shown once is whsec_ + standard base64 of the key. webhooks.md 2.1; the form the reference libraries decode.
D-M2-26-11 2026-10-15 A published webhook_out key is whpk_ + standard base64 of its DER-encoded SubjectPublicKeyInfo (RFC 8410); getWebhookKeys lists next, active and retiring keys ordered by state and kid with not_after only when set and the caching header of getJwks. The form of the example under getWebhookKeys; a receiver decodes it with any standard library.
D-M2-26-12 2026-10-15 docs/api/webhook-test-vectors.json is the contract of every verifier: derivation cases {master_hex, subscription_id, generation, derived_hex} (derived secrets only as derived_hex, because a derived value cannot carry the EXAMPLE marker) and signature cases {name, webhook_id, timestamp, body, ed25519_seed_base64, public_key, hmac_secrets, signature_header} with ed25519_seed_base64 and public_key as lists in signer order; generated once by TestWriteWebhookTestVectors from fixed seeds; every whsec_ value carries EXAMPLE. G10; two v1a signers need lists; no gitleaks allowlist entry.
D-M2-26-13 2026-10-15 event_type is not a column of webhook_deliveries: the delivery list reads it from the event log while the event is retained and omits it afterwards; WebhookDelivery.event_type left required in openapi.yaml. The core contract wins over the API document (spec gap, reported).
D-M2-26-14 2026-10-15 redeliverWebhook makes a pending row due now; a finished row becomes pending with last_error and response_code cleared, attempt kept and created_at now (a new retry window); the same webhook-id; an event no longer in the log is 409 conflict with reason: event_expired; the operation gains the 409 response. webhooks.md 4 point 4: a redelivery is a new delivery of the same event.
D-M2-26-15 2026-10-15 installApp with delivery.mode = webhook creates one subscription (url events_url, the subscribable event subscriptions of the installation, no HMAC, created by the installing owner) in the install transaction after webhook_out.manage, with its own audit entry; upgrades never change subscriptions; uninstall disables them; a suspended installation gets rows and sends nothing; app.resume makes the heads of its active subscriptions due (ResumeInstallation). 11 sections 4.1 to 4.3.
D-M2-26-16 2026-10-15 Keys webhook.max_consecutive_failures, webhook.retry_backoff_base, webhook.retry_backoff_cap (chapter 03) and webhook.max_pending_per_subscription, webhook.retry_max_age, webhook.secret_rotation_overlap, webhook.hmac_master_key_ref (chapter 11) in configkeys_webhook.go; webhook.delivery_timeout and webhook.allowed_hosts stay where M1-32 and M2-24 registered them; webhook.receiver_tolerance is no server key. Cross-checks webhook_backoff_order and webhook_retry_within_retention. New generated secret webhook_hmac_master_key (generator key, service server), resolved once at the start, so a missing key ends the start. Design note 15; a retry needs its event in the log.
D-M2-26-17 2026-10-15 Span webhook.deliver (CLIENT) with server.address, http.response.status_code and bothy.outcome from succeeded, throttled, gone, failed, blocked; logs webhook delivery failed (info) and webhook subscription state changed (warn) with the new attribute bothy.webhook_subscription.id, bothy.event.seq, bothy.event.type, bothy.reason and server.address; no new metric. signals.md 5.2 and 5.3; never path, query, body or secret.
D-M2-26-18 2026-10-15 Field codes not_subscribable, not_granted, hmac_not_enabled (and too_few_items, invalid for an empty or duplicate list); conflict reasons app_not_installed, one_time_secret (from the Idempotency-Key stage) and event_expired. Design notes 2 and 13; clients treat unknown codes like invalid.
D-M2-26-19 2026-10-15 Agent subscriptions are not delivered: webhook_out.deliver answers forbidden subject_kind_not_allowed for every principal other than an app, and listAgentWebhookSubscriptions and createAgentWebhookSubscription stay under later without a ticket. Agent delivery needs agent.invoke and a fresh invocation token per delivery through M2-08 (plan gap, reported).
D-M2-26-20 2026-10-15 listWebhookDeliveries lists newest first by (event_seq, id) descending with a cursor bound to subscription and status filter. The latest attempts matter most to an owner.
D-M2-26-21 2026-10-15 Names that differ from the brief: the methods of apps.Webhooks take the caller actor.Actor like apps.Apps; PublicKeys authorizes instance.discovery.read like JWKS; the delivery decision lives in Authorizer.authorizeDelivery with the gate domain.DeliveryGate and the requirement marker domain.Delivery; the egress refusal reaches the pure classification as domain.BlockedError; signing keys and the job runtime start after the apps step and are bound later (wire.Apps.BindKeys, BindJobs). A scope removed through setAppScopes also removes the event subscription that needs it (M2-24), so the delivery ends dead with not_subscribed before the rights check; with the subscription kept the rights check ends it with not_authorized (TestDeliveryScopeRemovedIsDead covers both). The merged code wins over the brief (D-W-100); 02 start order and the consent rules of M2-24 stay.
D-M2-26-22 2026-10-15 The webhook scenarios live in services/server/test/appsworld (scenarios_api.go, scenarios_delivery.go) and the packages app, adapters/httpapi and adapters/postgres of apps call them, so each package is measured by the scenarios that run through it (the pattern of D-M2-24-23). The golangci exclusion of gocognit and goconst for test code covers test/ as well as *_test.go, matching sonar.test.inclusions. make coverage measures each package only by its own tests; code under test/ is test code (02 Code-Layout), which the analysis already treats so.
D-M2-26-23 2026-10-20 Port onto main from the integration squash of int/m1-wave23: the twelve own commits of the ticket from the commit after 1cab9e641 (that one ported webhook_out.transition and R-AZ-27 from main, which main has). The German chapter edits go into the English chapters 02 and 11, the ADRs 0016 and 0051, webhooks.md, the threat model and the signals; code comments name the English chapter files; the migrations become 00037 and 00038 after 00034 of M1-60 and 00035/00036 of M3-13 (comments changed, checksums locked again); the example HMAC secrets of the test vectors are whsec_EXAMPLEvect1 and whsec_EXAMPLEvect2, shorter than the secret format hook of main allows to look like a Stripe secret, and the non-ASCII sample bodies use no German words; the vectors are generated again; config-schema.json and the configuration reference (with the secret webhook_hmac_master_key) are generated again. INT-W23/1 to /3 concern the integration order and other tickets. The PR lane ports only the ticket and the integration fixes it needs; vet, migrations lint, the configuration check, the language and secret format gates and the tests of the touched server packages pass locally.