M2-26: Outgoing webhooks after Standard Webhooks¶

Status: ready | Depends on: M1-56, M2-01, M2-24 | Milestone: M2

Goal¶

After this ticket an owner subscribes an installed app to event types (createWebhookSubscription, webhook_out.manage, step-up), and the worker delivers every matching durable event from the event log to the app's HTTPS address as one Standard Webhooks POST (ADR 0016): the body is the event envelope of ADR 0052 with the delivery field sync_epoch, the headers are webhook-id (the event ID), webhook-timestamp and webhook-signature with v1a (Ed25519 with the server key of purpose webhook_out, public keys at getWebhookKeys) and, per subscription, v1 (HMAC-SHA256 with a secret derived from a master key and never stored). Delivery runs as the first production consumer webhooks of the M2-01 frame and the job apps.webhook_deliver: strictly in seq order per subscription, with backoff and full jitter, Retry-After honoured for 429, 502, 503 and 504, a subscription switched off by 410, by webhook.max_consecutive_failures or by a delivery that stays undelivered for webhook.retry_max_age, and a check webhook_out.deliver with the app's current grants and scopes before every attempt, so that a revoked right ends the delivery as dead without any request (AZ-M49). Every connection goes through the egress client, so a target that resolves to a private address is never contacted. Gate G10 holds the signatures against committed test vectors, the Standard Webhooks reference library (v1) and Ed25519 of the Go standard library (v1a). This is the plan's "webhooks" part of M2 (P-18: apps and webhooks use the same public API) and the server half of threat T-PL-01 ("webhooks only to allowed targets").

Read first¶

docs/PLAN.md: section 3 "Way of working per ticket"; section 8 "M2 Agent platform, extensions and terminal" with "Goal" (P-18), "Scope" (webhooks from 06 and docs/api/webhooks.md; ADR 0016, 0051), "Out of scope" (user interface M4), "Exit criteria" points 3 (AZ-M49), 4 (G10, M2-26) and 10, "Test strategy" (contracts G10; chaos "duplicate delivery"), "Red team" ("webhooks to private targets; exceeding the scopes of an app"; webhook targets as boundaries), "Risks"; block "agents and extensions" rows M2-01, M2-07, M2-08, M2-24, M2-25, M2-26, M2-27, M2-30 (uses the test vectors of this ticket), M2-49, M2-50; rows M6-08 (console views), M7-04 (bothy restore finalize), M7-05 (retention jobs), M7-08 (maintenance mode); section 5 "Test strategy" (row "Contract": G10 in the integration tests); section 19 P-09, P-10, P-11, P-13, P-18.
Dependencies: the header names M1-56 (message.created, the chat structure source with spaces, channels and threads), M2-01 and M2-24 (D-W-103). Implied through them and therefore not in the header: M1-55 (threads), M1-53 (channels), M1-52 (spaces), M1-51 (table agents for the foreign key webhook_subscriptions.agent_id), M1-46, M1-36, M1-31, M1-44, M1-35, M1-34, M1-33, M1-32 (instance.Keys, signing keys, rotateSigningKey), M1-29, M1-28 (authorize, deferred entries), M1-27, M1-26, M1-25, M1-24 (audit), M1-23, M1-22 (egress client), M1-20 (idempotency, one-time secrets), M1-18 (HTTP chain), M1-17 (route generator, api/operations.yaml), M1-16 (event log, sync.Event, envelope), M1-15 (instance.State, sync_epoch), M1-14 (jobs), M1-13 (unit of work), M1-12, M1-11, M1-10, M1-09 (secret provider, secrets catalogue), M1-08, M1-07, M1-06, M1-05, M1-03, M1-02, M1-01. Used if merged: M1-63 (mapping of a stored event to the resource of its receive action), M1-69 (canary scenario), M1-70 (tools/lint/sql_visibility.yaml), M1-71 (services/server/test/harness with BuildMatrixWorld).
docs/api/webhooks.md completely: 1 "Subscribe", 2 "Delivery" (table of headers, signed content, sync_epoch), 2.1 "Signing keys", 3 "Verification by the receiver", 4 "Retries and disabling", 5 "Order and duplicates", 6 "Delivery security", 7 "Event types for apps and agents" (generated table and the list "Not subscribable"), 9 "Configuration values", "Sources" W1 and W2.
docs/spec/11-extensions.md: 1 "Principles" point 4; 2 "Agent or app"; 3.1 "Format" (events, delivery.mode, delivery.events_url); 3.2 "Validation rules" points 1 to 3; 4.1 "Installation" (the diagram writes webhook_subscriptions in the install transaction); 4.2 "Update"; 4.3 "Suspension and uninstallation" (row "Deliveries"); 5 "Scopes and grants"; 6 "Isolation and limits" (rows "Delivery quota", "Timeouts", "Suspension" and the last paragraph on "gap"); 7 "Outgoing webhooks" completely; 14 R-EX-06, R-EX-07; 16 "Configuration values of this chapter".
docs/spec/03-data-model.md (core contract, read only): 4.11 webhook_subscriptions and webhook_deliveries (every column, check, foreign key, index, retention class); 4.1 server_signing_keys (purpose webhook_out, alg = 'EdDSA' only, states, not_after); app_installations (granted_scopes, event_subscriptions, state); event_log (audience_principal_id, receive_action, retention); audit_events (action is an action id of 05); 1 principle 9; 6.3 "Lock order"; 6.7 "Sync epoch and restore" item 1; 8 R-DM-06, R-DM-17; 9 "Configuration values of this chapter" (webhook.allowed_hosts, webhook.max_consecutive_failures, webhook.retry_backoff_base, webhook.retry_backoff_cap).
docs/spec/04-sync-protocol.md (core contract, read only): 2.2 "Envelope" (row sync_epoch, "Delivery fields are created per delivery and are never in the event log", payload rules 1 to 5), 7.5 "Sync epoch" item 4, 7.6 "Consumers in the worker (push, webhooks)", 8.1 "Basic rule" (rights at delivery time), 9 "Delivery guarantee", 10 "Ordering", 15.6 "Webhook as transport" (agent subscriptions, see "Out of scope").
docs/spec/05-actions.md (core contract, read only): 1 rules 1, 2, 3, 5 and 10; 2.2 "Target of a resource"; 2.3 "Effective level" point 5; 2.4 (columns "Owner", "Step-up", "Audit"); 2.5 "Decision"; 2.6 "Scopes for apps" (events:subscribe, messages:read, the scope-free metadata actions, webhook_in.post); 2.7 "Effect (effect)" (webhook_out.deliver has read); 4.11 "Apps and webhooks" rows webhook_out.read, webhook_out.manage, webhook_out.redeliver, webhook_out.deliver, webhook_out.transition (system action, R-AZ-27), app.suspend, app.resume, app.uninstall; 6 "Durable event types and receive action" (last sentence: webhooks use webhook_out.deliver); 8 R-AZ-01, R-AZ-03, R-AZ-04, R-AZ-11, R-AZ-15 (system actions in the catalogue) and R-AZ-27 (webhook_out.transition for the automatic subscription transitions).
docs/spec/07-permissions.md: 2.10 "Apps"; 4.1 "Decisions and reasons" (scope_missing, app_not_installed, subject_kind_not_allowed, not_self, no_grant); 4.2 "Flow" (deferred entries; filter mode for the receive check of webhooks; allow audit written by the use case); 4.3 "Visibility"; 8.1 "Delivery filter" (row "Outgoing webhooks"); 10 "Examples" and 11.1 "Test matrix" (world, G-6, row AZ-M49); 11.2 "Required tests" AZ-P1, AZ-P2, AZ-P4, AZ-P13.
docs/spec/06-api.md: 4 "Authorization in the API" points 4 and 7; 5 "Errors" (points 3, 4, 7 and the generated table: validation_failed, webhook_host_not_allowed, step_up_required, forbidden, scope_missing, not_found, conflict, idempotency_key_reuse); 6 "Pagination"; 7 "Idempotency" (row "POST with a one-time secret in the response (enrollment code, API token, later webhook secrets)", R-API-26); 12 "Webhooks"; 15 "Contract tests and gate" (G10); 16 "Endpoints" groups "Discovery" (getWebhookKeys) and "Webhooks"; 19 R-API-14, R-API-22, R-API-26.
docs/spec/02-architecture.md: 3.2 "Roles of a process"; 4.2 "Module list" (row apps); 4.3 "Ports, events and sources per module" (apps port Webhooks, instance port Keys, sync port Consumers); 4.4 "Allowed dependencies" (row apps: instance, identity, chat, agents, messaging, sync); 4.5 "Cross-cutting ports (internal/kernel)" (kernel/authorize filter mode); 4.6 "Actions per module" (row apps); 5 "Layers within a module"; 8.2 "Lock order"; 8.4 "Idempotency in the Unit of Work"; 8.5 "What never happens in a transaction"; 11.5 "Worker consumers (push, webhooks)"; 12.1 "Usage", 12.2 "Queues" (delivery), 12.3 "Job catalogue" (row apps.webhook_deliver), 12.4 "Rules for jobs" (rules 1 to 4); 13 "Configuration loading" items 3 to 6; 14.2 "Deadlines"; 14.3 "Retries".
docs/spec/19-security.md: 4 SEC-CORS-2; 8 SEC-NET-1 to SEC-NET-5; 9 SEC-LOG-1 to SEC-LOG-4; 10 "Secrets" (row of the private signing keys, SEC-SEC-1, SEC-SEC-3); 12 SEC-AUD-5; 14 SEC-T09, SEC-T10.
docs/spec/20-privacy.md: 2 "Data categories per table" rows webhook_subscriptions and webhook_deliveries (class transient, retention.webhook_deliveries).
docs/api/openapi.yaml: operations getWebhookKeys, listWebhookSubscriptions, createWebhookSubscription, updateWebhookSubscription, deleteWebhookSubscription, listWebhookDeliveries, redeliverWebhook, listAgentWebhookSubscriptions, createAgentWebhookSubscription, installApp, uninstallApp, getJwks (pattern for getWebhookKeys); schemas WebhookKeys, WebhookKey, WebhookSubscription, WebhookSubscriptionCreate, WebhookSubscriptionUpdate, WebhookSubscriptionCreated, WebhookSubscriptionList, WebhookDelivery, WebhookDeliveryStatus, WebhookDeliveryPage, EventEnvelope, App, AppInstallation; parameters WebhookId, WebhookTimestamp, WebhookSignature, AppIdPath, SubscriptionIdPath, DeliveryIdPath, Cursor, Limit, IdempotencyKey; responses BadRequest, Unauthorized, Forbidden, NotFound, Conflict, UnprocessableEntity, TooManyRequests; the section webhooks: (every entry carries x-authz-action: webhook_out.deliver, x-receive-action and the responses 2xx, 410, 429 "same for 502, 503 and 504").
docs/spec/actions.yaml on the base branch (generated catalogue of ADR 0049): the entries webhook_out.* (delivery_check: true on webhook_out.deliver) and the events: list with receivers: agent for the agent-only types.
ADRs: 0016 (all, especially "Confirmation"), 0051 (items 4 and 5, "Confirmation": "a permission revocation between round and job prevents the sending"), 0052, 0053, 0059, 0049, 0066, 0022, 0013, 0007.
docs/DECISIONS.md: D-EX-02, D-API-09, D-API-16, D-K-B15, D-K-D14, D-K-D30, D-K-D32, D-W-23, D-W-25, D-W-100, D-W-104 (R-AZ-27); section 10 "Rulings of the build milestones" (row format D-<Ticket>-<n>).
docs/security/threat-model.md: 2 TB6; 5.9 "Worker and jobs" (row "Webhook to a wrong target"); 5.11 "Apps and webhooks"; T-PL-01, T-NW-01, T-DS-01. docs/RISKS.md: RK-04 (SSRF over webhooks), RK-20 (v1a without receiver libraries, HMAC with test vectors). docs/OPEN-QUESTIONS.md has no item for this ticket.
docs/observability/signals.md: 5.2 "Spans" (row webhook.deliver), 5.3 "Attributes in spans and logs", 7 "Logs"; row bothy.egress.blocked. No metric of signals.md belongs to this ticket; docs/observability/alerting.md has no webhook alarm.
docs/operations/configuration.md: section 4 rows webhook.allowed_hosts, webhook.max_consecutive_failures, webhook.retry_backoff_base, webhook.retry_backoff_cap, webhook.delivery_timeout, webhook.max_pending_per_subscription, webhook.retry_max_age, webhook.secret_rotation_overlap, webhook.receiver_tolerance (receiver only, not a server key), egress.allowed_hosts, egress.timeout, egress.max_response_bytes, sync.event_retention, consumer.batch_size, jobs.queue.delivery.max_workers, agent.jwks_max_age, secrets.cache_ttl, retention.webhook_deliveries, database.tx_retry_max, database.lock_timeout; section 5 "Secrets" (catalogue format).
M1 and M2 briefs whose interfaces you use: docs/briefs/M2-01-worker-event-consumers.md (sync.ConsumerName, sync.ConsumerWebhooks, sync.ConsumerHandler.HandleEvent, sync.Consumers.Register, Design note 2 "handler contract", note 5 "gaps" (PlanRound, LostFrom, LostTo), note 6 "no durable events in a round"), M1-32 (instance.Keys with Signer, VerificationKey, PublicKey, JWKS; instance.Purpose; states; getJwks with Cache-Control and CORS), M1-28 ("Public contract in kernel/authorize", authorize.Filter, "Requirement table (domain/requirements.go)", "Deferred entries", "App scopes (domain/scopes.go, 05 section 2.6)", "Visibility (domain/visibility.go, 07 section 4.3 per resource kind, ruling D4)", TestDeferredActionsGolden), M1-25 (VerifyRegistrations), M1-24 (auditing.Recorder.Record, auditing.Entry, auditing.ID, auditing.Code, auditing.Int, auditing.Bool, auditing.List), M1-22 (egress.New, egress.Options, egress.RedirectNever, egress.HostPolicy, egress.Deps, egress.ErrBlocked, egress.BlockedError), M1-20 (idempotency stage, idempotency.MarkOneTimeSecret), M1-18 (problem writer with reason and errors[], cursor codec in platform/httpserver, httpservertest), M1-17 (api/operations.yaml, route generator), M1-16 (sync.Event, sync.DeliveryFields, Event.Envelope, minimum read), M1-15 (instance.State.Load, Record.SyncEpoch, platform/buildinfo.Version), M1-14 (jobs.Register, jobs.HandlerOptions, jobs.InsertOptions.ScheduledAt, jobs.Inserter.InsertTx, jobs.ValidateArgsType), M1-13 (txn.UnitOfWork.Do, txn.Key), M1-12 (pgtest.Main, pgtest.New, migration numbering, grants), M1-10 (internal/platform/otel/attr, TestAttrKeysEqualCollectorAllowlist), M1-09 (secrets.Provider.Resolve, internal/platform/secrets/stack catalogue, generator key), M1-07 and M1-08 (config.Key, chapter constants, make config-reference). The brief of M2-24 for apps, app_installations, the manifest model, installApp and uninstallApp.
Code on the base branch (merged names win, D-W-100), read with git show before changing anything: services/server/internal/apps (README and everything M2-24 merged), internal/sync (port with M2-01's Consumers), internal/instance (keys, state, adapters/httpapi with getJwks), internal/authz (domain/requirements.go, domain/scopes.go, domain/visibility.go, domain/decide.go, the deferred list), internal/kernel/{authorize,actor,auditing,idempotency,eventlog,txn,ids,clock,errs}, internal/platform/{egress,jobs,secrets,otel,httpserver,pg}, internal/wire (config.go, configchapters.go, configchecks.go, egress.go, the role assembly), services/server/architecture/modules.yaml, services/server/api/operations.yaml (entries of the group "Webhooks" under later with ticket M2-26, and getWebhookKeys), services/server/migrations/, .gitleaks.toml (rule standard-webhooks-secret, allowlist stopword example), infra/otel/collector.yaml.

Fixed requirements that apply¶

Owner's fixed build rules (apply verbatim):

No MVP. New from scratch and directly in the final state; stages are only the order.
Strictly modular: clear module and service boundaries, dependencies only through defined interfaces, no cross access.
Documentation in code: every public module, class, function and interface with a docstring/doc comment (purpose, parameters, return value, errors, example where useful); comments explain the why, not the what. API documentation is generated from code (OpenAPI from the routes, dartdoc, Python API docs in MkDocs).
Documentation in the repo: README per module/service, ADR for every architecture decision, CHANGELOG per release, keep the documentation site current. A change without matching documentation is not done.
Quality gates (SonarQube, linters, type checking, security scanners, test coverage) block the merge.

Rules for every brief:

Test first: write the failing test, run it and keep the red output for the report, then make it green, then clean up.
No en dash and no em dash in any file you write; the text-rules pre-commit hook rejects them. Use a hyphen or rephrase.
White label: examples and test data use example.com and the people Alex, Sam and Robin; no company, host or personal names, no paths under a home directory.
Limits, windows and thresholds only through named configuration keys listed in docs/operations/configuration.md; code comments and docs name the key, never the number.
Specification first: a change to behaviour, API, event, right or configuration updates docs/spec/ or docs/api/ in the same change. The core contract (docs/spec/00-glossary.md, 03-data-model.md, 04-sync-protocol.md, 05-actions.md) is not edited by this ticket.

Product requirements implemented here:

Tables webhook_subscriptions and webhook_deliveries exactly as 03 4.11, owned and accessed only by apps (02 4.2; 03 principle 12): every column, the check "exactly one of app_installation_id and agent_id", hmac_enabled OR hmac_generation = 0, the state and status checks, the foreign keys with their actions, unique (subscription_id, event_seq), the indexes (state), (agent_id) WHERE agent_id IS NOT NULL and (next_attempt_at) WHERE status = 'pending'; retention classes config and transient (20).
A subscription is created, changed, paused, resumed, rotated and deleted only by an owner in a session with step-up, audited (webhook_out.manage; 05 4.11, R-AZ-03); lists and delivery history need webhook_out.read (owner); redeliverWebhook needs webhook_out.redeliver (owner, audited, no step-up). An invisible resource answers 404 exactly like an unknown ID (05 rules 1 and 2, R-AZ-01).
The address is https and its host is in webhook.allowed_hosts, otherwise 400 webhook_host_not_allowed (R-DM-17; webhooks.md 1 point 2; 06 5); event_types is a subset of the installation's event_subscriptions (03; 11 7 row "Subscription").
Delivery comes from the event log through the worker consumer webhooks and the job apps.webhook_deliver in queue delivery (02 11.5, 12.3; ADR 0051; 11 6 "Deliveries come from the event log, not from a copy"); no network call inside a transaction (02 8.5, R-AR-07).
Wire format after Standard Webhooks: one POST per event; body = the envelope of the event (04 2.2, ADR 0052) with the delivery field sync_epoch and no JSON null (ADR 0053); webhook-id = event ID, unchanged across retries and redeliveries; webhook-timestamp = Unix seconds of this attempt; webhook-signature = space-separated v1a,<base64> (Ed25519, server key of purpose webhook_out) and, when hmac_enabled, v1,<base64> (HMAC-SHA256); signed content {webhook-id}.{webhook-timestamp}.{body} over the body bytes as sent (webhooks.md 2; 11 7; ADR 0016).
Public keys of purpose webhook_out in the states next, active and retiring at GET /api/v1/instance/webhook-keys (getWebhookKeys, instance.discovery.read, public, rl.public_auth) in the format whpk_<base64> (webhooks.md 2.1 point 1; 11 7 last paragraph; SEC-CORS-2). The JWKS (getJwks) never lists webhook_out keys.
The v1 secret is never stored: HMAC-SHA256(master key, "webhook-v1:" || subscription_id || ":" || generation) with the master key from the secret provider, shown once as whsec_<base64>; rotate_hmac_secret raises hmac_generation and the server signs with the old and the new generation for webhook.secret_rotation_overlap (webhooks.md 2.1 point 3; 03 hmac_generation, hmac_rotated_at; 03 principle 9). Responses with a secret are one-time secrets for idempotency (06 7, R-API-26).
Strict order per subscription: the next event goes out only after the previous one is finished; last_delivered_seq is the highest seq accepted with 2xx (webhooks.md 5; 11 7 row "Order"; 03).
Outcomes: only 2xx is success; 410 switches the subscription off (disabled); 429, 502, 503 and 504 throttle and honour Retry-After; 3xx is a failure and redirects are never followed; other statuses, timeouts and connection errors are failures retried with exponential backoff and full jitter between webhook.retry_backoff_base and webhook.retry_backoff_cap until webhook.retry_max_age, then the delivery is dead and the subscription disabled; after webhook.max_consecutive_failures failures in a row the subscription is disabled and audited; response bodies are never stored (webhooks.md 4, 5 point 3; 11 7; openapi webhooks: responses).
Before every attempt webhook_out.deliver decides with the app's current grants and scopes (receive action of the event type for the app, filter mode, no denial in the audit); without access the delivery is dead with last_error = not_authorized, without a request and without counting as a failure (05 4.11, R-AZ-11; 07 4.2, 8.1; webhooks.md 4 point 3; 11 7 row "Check"; ADR 0051 "Confirmation").
SSRF protection: only https, only hosts of webhook.allowed_hosts, the egress client checks the address after name resolution at connection time and connects only to the checked address, loopback, link-local and private addresses are refused unless egress.allowed_hosts has a matching entry with allow_private, no redirects, time limit webhook.delivery_timeout per attempt (webhooks.md 6; 11 7 row "SSRF protection"; SEC-NET-1 to SEC-NET-5).
A paused subscription continues where it stood; an event that is no longer in the log when its turn comes switches the subscription to disabled with the hint "gap" instead of being skipped silently, and so do events the consumer webhooks itself lost to retention (11 6 last paragraph, R-EX-07; ADR 0051 Consequences). Above webhook.max_pending_per_subscription open deliveries a subscription is paused and the owner learns it (11 6 row "Delivery quota").
app.suspend pauses deliveries and keeps their state; app.uninstall sets the app's subscriptions disabled (11 4.3). Installing an app whose manifest has delivery.mode = webhook creates its subscription in the install transaction (11 4.1).
Logs, spans and audit carry no URL path or query, no body, no secret and no content (SEC-LOG-1 to SEC-LOG-4, SEC-SEC-3, SEC-AUD-5); the span webhook.deliver (CLIENT, worker) carries server.address, http.response.status_code and bothy.outcome (signals.md 5.2).
Gate G10: committed test vectors; a delivery in an integration test verified with the Standard Webhooks reference library (v1) and with Ed25519 of the standard library (v1a) (06 15; ADR 0016 "Confirmation").

Scope¶

All paths under services/server/ unless noted. Where a merged M1 or M2 package already has a compatible name, keep it and list the difference in the report (D-W-100).

Create:

migrations/<next free number>_webhook_subscriptions.sql and migrations/<next free number + 1>_webhook_deliveries.sql (goose, numbering rule of M1-12): the tables of requirement 11; GRANT SELECT, INSERT, UPDATE, DELETE on both to bothy_app. Then make migrations-lock.
internal/apps/webhooks.go (port, standard library and internal/kernel only): type Webhooks interface with CreateSubscription(ctx context.Context, cmd CreateSubscription) (SubscriptionCreated, error), UpdateSubscription(ctx context.Context, cmd UpdateSubscription) (SubscriptionCreated, error), DeleteSubscription(ctx context.Context, id ids.ID) error, ListSubscriptions(ctx context.Context, appID ids.ID) ([]Subscription, error), ListDeliveries(ctx context.Context, q DeliveryQuery) (DeliveryPage, error), Redeliver(ctx context.Context, deliveryID ids.ID) (Delivery, error) and ResumeInstallation(ctx context.Context, installationID ids.ID) error (for app.resume, Design note 14); the DTOs Subscription, SubscriptionCreated (subscription plus optional one-time HMACSecret), CreateSubscription, UpdateSubscription, Delivery, DeliveryQuery, DeliveryPage; the constants of the subscription states and delivery statuses of 03 and the last_error codes of Design note 6.
internal/apps/domain/subscription.go: pure state rules NextSubscriptionState(current SubscriptionState, ev SubscriptionEvent) (SubscriptionState, error) for the events of Design note 5; ValidateTarget(raw string, allowed []AllowedHost) (Target, error) (Design note 9); ValidateEventTypes(types, granted []string) error; AppSubscribable(eventType string) bool over the set of Design note 10.
internal/apps/domain/delivery.go: Classify(statusCode int, transportErr error) Outcome (Design note 6), Backoff(base, cap time.Duration, failedAttempts int, rnd func(n int64) int64) time.Duration (Design note 7), ParseRetryAfter(header string, now time.Time) (time.Duration, bool) (Design note 7), NextAttempt(o Outcome, attempt int, retryAfter time.Duration, hasRetryAfter bool, createdAt, now time.Time, cfg RetryConfig, rnd func(n int64) int64) (next time.Time, dead bool) combining both with the retry deadline (RetryConfig holds webhook.retry_backoff_base, webhook.retry_backoff_cap and webhook.retry_max_age).
internal/apps/domain/signature.go: SignedContent(webhookID ids.ID, ts int64, body []byte) []byte, DeriveHMACKey(master []byte, subscriptionID ids.ID, generation int) []byte, FormatSecret(key []byte) string (whsec_), ParseSecret(s string) ([]byte, error), SignatureHeader(v1a [][]byte, v1 [][]byte) string, HMACSignature(key, content []byte) []byte (Design note 11).
internal/apps/app/webhooks.go (use cases of the port), internal/apps/app/consumer.go (the sync.ConsumerHandler of the consumer webhooks, Design note 3), internal/apps/app/deliver.go (Deliver(ctx context.Context, subscriptionID ids.ID) error, the attempt of Design note 4) and outgoing ports in internal/apps/app/ports.go: WebhookStore (persistence), DeliveryScheduler { Schedule(ctx context.Context, subscriptionID ids.ID, at time.Time) error } (inserts a job in the ambient unit of work), WebhookSender { Send(ctx context.Context, req SignedRequest) (SendResult, error) }, HMACMasterKey { Key(ctx context.Context) ([]byte, error) }.
internal/apps/adapters/postgres/queries/webhooks.sql (sqlc) with the adapter implementing WebhookStore.
internal/apps/adapters/jobs/webhook_deliver.go: args WebhookDeliver struct { SubscriptionID ids.ID } with Kind() = "apps.webhook_deliver", registered with jobs.HandlerOptions{Queue: "delivery"}; the handler calls Deliver; the DeliveryScheduler over jobs.Inserter.InsertTx with jobs.InsertOptions.ScheduledAt.
internal/apps/adapters/webhookhttp/sender.go: WebhookSender over one egress.Client built with Purpose: "webhook", Hosts from webhook.allowed_hosts, Redirects: egress.RedirectNever, Timeout = webhook.delivery_timeout; it sets the headers of requirement 15 and User-Agent: chat-server-webhooks/<server version> (webhooks.md 2, version from platform/buildinfo), reads only the status and Retry-After, discards the body, and opens the span webhook.deliver.
internal/apps/adapters/hmackey/master.go: HMACMasterKey over secrets.Provider.Resolve of webhook.hmac_master_key_ref, hex-decoding the value like every key of generator key (M1-09 D5), held in the redacting secret type.
internal/apps/adapters/httpapi/webhooks.go: the strict-server methods of the six enabled operations of the group "Webhooks".
internal/instance/adapters/httpapi: the strict-server method GetWebhookKeys next to getJwks.
internal/wire/configkeys_webhook.go: webhookConfigKeys() with the keys of Design note 15 and a constant chapter11 = "11-extensions.md" (if a parallel ticket added it, keep one).
docs/api/webhook-test-vectors.json (Design note 12) and a row for it in docs/api/README.md.
Tests of the section "Tests" with their fixtures.

Change:

internal/sync (port of M1-16 and M2-01): DeliveryFields gains SyncEpoch ids.ID unless M1-16 already has it, and Event.Envelope writes the member sync_epoch of EventEnvelope only from it (04 2.2: only in webhook deliveries); a read port type LogReader interface { EventBySeq(ctx context.Context, seq int64) (Event, bool, error); MinRetainedSeq(ctx context.Context) (int64, error) } implemented in internal/sync/adapters/postgres (reuse the minimum read of M1-16); type ConsumerLossHandler interface { HandleLost(ctx context.Context, from, to int64) error } in the port and its call in the round of internal/sync/app (Design note 3 f); README of sync.
internal/instance (port of M1-32): Keys gains Signers(ctx context.Context, p Purpose) ([]Signer, error) (the active key first, then every retiring key whose not_after lies after now, never next or retired) and PublicKeys(ctx context.Context, p Purpose) ([]PublicKey, error) (states next, active, retiring); a pure FormatWebhookPublicKey(ed25519.PublicKey) string (Design note 11); README of instance.
internal/kernel/authorize (resource.go): Resource gains DeliveryOf Action (receive action of the event to deliver) and Audience ids.ID (audience principal of a personal event, zero otherwise), both used only by webhook_out.deliver.
internal/authz: replace the deferred entry of webhook_out.deliver with the requirement of Design note 8; remove webhook_out.deliver from the golden list of TestDeferredActionsGolden; README of authz (deferred list, delivery check).
internal/apps/domain (M2-24 rules.go): the address rule of Design note 9, with a new row in TestValidateManifest_Rules: events_url on a listed host with an unlisted port gives webhook_host_not_allowed.
internal/apps use cases of M2-24: installApp creates the manifest subscription and uninstallApp disables the subscriptions in their transactions (Design note 14); if app.resume exists in the merged code, it calls ResumeInstallation in its transaction.
internal/platform/secrets/stack: catalogue entry webhook_hmac_master_key (generated, generator key, service server), so bothy secrets init creates it.
internal/platform/otel/attr: constant for bothy.webhook_subscription.id; infra/otel/collector.yaml: the key in redaction/allowlist.allowed_keys ("Bothy attributes").
internal/wire: configKeyGroups gains webhookConfigKeys; configChecks gains webhook_backoff_order and webhook_retry_within_retention (Design note 15); TestConfigSchemaBuilds lists the new keys; the role assembly builds the webhook egress client, the sender, the master-key adapter and the apps webhook service in every role, registers the job kind in every role (the api role inserts jobs from redeliverWebhook and subscription changes), and in the roles worker and all registers the handler for sync.ConsumerWebhooks with Consumers.Register before the runner starts (replace the comment M2-01 left there).
api/operations.yaml: move createWebhookSubscription, deleteWebhookSubscription, getWebhookKeys, listWebhookDeliveries, listWebhookSubscriptions, redeliverWebhook and updateWebhookSubscription to enabled (sorted); keep listAgentWebhookSubscriptions and createAgentWebhookSubscription under later with milestone: M2 and without the field ticket (Out of scope); rerun the route generator.
docs/api/openapi.yaml: WebhookDelivery.event_type no longer in required, with the description of Design note 13; WebhookKey.public_key description names the encoding of Design note 11; updateWebhookSubscription and redeliverWebhook gain the response '409': $ref: '#/components/responses/Conflict' (Design notes 2 and 13); descriptions of createWebhookSubscription and updateWebhookSubscription name the one-time secret rule. Then make api-lint, make api-checks, make api-tables and the route generator.
docs/api/webhooks.md: 1 (validation codes of Design note 9), 2.1 (signing set and key encoding of Design note 11, HMAC encoding, a new 2.2 "Test vectors" naming the file), 4 (statuses, last_error codes, throttles not counted, Retry-After rule, failed and dead, redelivery and 409 conflict with reason: event_expired), 5 (rows while paused and disabled, gap rule), all as decided in the Design notes.
docs/spec/11-extensions.md: 6 rows "Delivery quota" and the paragraph on "gap", which names both detection points, the head of a subscription and the lost range of the consumer round (Design notes 3 and 5), 7 rows "Check" and "Success" (503 added; "skipped" means dead with not_authorized, not counted), 16 new row webhook.hmac_master_key_ref; then make config-reference.
docs/spec/02-architecture.md: 4.3 port names (apps.Webhooks, sync.LogReader, instance.Keys.Signers), 12.3 row apps.webhook_deliver (triggers of Design note 4), 13 item 4 (the two cross checks).
docs/operations/configuration.md section 5: row webhook_hmac_master_key (generated, service server, webhook.hmac_master_key_ref); section 4 regenerated.
docs/observability/signals.md: 5.2 row webhook.deliver with the closed bothy.outcome list of Design note 16; 5.3 row bothy.webhook_subscription.id.
docs/security/threat-model.md 5.11: the rows of Design note 17.
docs/adr/0016-webhooks-per-standard-webhooks.md: "Confirmation" names the tests of this ticket and the vector file; the decision stays; docs/adr/0051-worker-consumers-of-the-event-log-with-a-cursor.md: in the last bullet of "Consequences" "last_delivered_seq detects the gap" becomes the two detection points of Design notes 3 f and 5 e, and "Confirmation" names the revocation test and TestConsumerLossDisablesSubscriptions; make adr-index.
READMEs of apps, sync, instance, authz; services/server/CHANGELOG.md ("Unreleased"); docs/DECISIONS.md section 10 rows D-M2-26-1 to D-M2-26-20 (Design note 18).

No durable or ephemeral event type, no action id and no rights version change is added by this ticket. This ticket creates no new top-level Makefile target.

Out of scope¶

Webhooks as agent transport (09 4.2, 04 15.6, webhook_subscriptions.agent_id, operations listAgentWebhookSubscriptions and createAgentWebhookSubscription): every delivery to an agent needs agent.invoke with I2 and a freshly issued invocation token per delivery (R-SY-07) through runs.Delivery of M2-08, which is not in this ticket's dependency closure. The table carries the column and its check as 03 defines them; webhook_out.deliver answers forbidden subject_kind_not_allowed for agent principals until the owning ticket replaces it; the report names the plan gap.
The operations suspendApp and resumeApp themselves (no ticket in the plan; reported as a plan gap); this ticket makes delivery honour suspended and offers ResumeInstallation.
Incoming webhooks native and alertmanager (M2-27); commands and interactions of apps, the events command.invoked and interaction.created to apps (M2-25); the manifest, installation and scopes (M2-24).
The TypeScript webhook verifier and the SDK handling of the vectors (M2-30; M2-29 for Python); the reference apps (M2-49, M2-50).
Console and desktop views for subscriptions and deliveries (M6-08).
bothy restore finalize setting last_delivered_seq to the head and a new sync_epoch (M7-04); the maintenance mode exemption of consumer webhooks and queue delivery (M7-08); retention jobs of the class transient (M7-05; Design note 6 states the rule they must keep).
The worker consumer frame, cursor, metrics and alarm (M2-01; this ticket adds only the loss call of Design note 3 f); the egress client itself (M1-22); signing key rotation (M1-32).
Changes to the core contract.

Boundaries¶

A new migration needs its line in migrations.sha256: run make migrations-lock after creating the file and before make check; never edit a merged migration (M1-01, ADR 0007).
Work in a dedicated git worktree on the branch feat/M2-26-webhooks-outgoing; one writer per worktree.
Conventional Commits with the module as scope (for example feat(apps): ...) and a DCO sign-off on every commit (git commit -s); squash into meaningful commits before the merge (docs/quality/push-discipline.md).
Never push to main, never tag and never create remote repositories; the only remote steps are the branch push and the pull request of AGENTS.md.
Do not modify files outside the scope above, except the CHANGELOG, README and docs/DECISIONS.md entries named in this brief.
Follow the host rules supplied by the launcher (ports, machine rules).
Tests bind only 127.0.0.1 and port 0; PostgreSQL runs only as a throwaway container on a random port through pgtest (P-09, P-10). Without a Docker-compatible runtime the integration tests fail; they never skip.
Stop only processes you started, by PID. Temporary directories only via mktemp -d, removed at the end.
Never read or print secrets. No network calls except package downloads for pinned tools and Go modules.
Never lower a threshold, disable a gate or delete a test.
If the core contract lacks something this ticket needs, stop with Verdict: BLOCKED <reason> instead of building a deviation (P-11).
This brief was written before milestone M1 closed (D-W-100). Where the merged code of an M1 ticket names a package, type, function, file or make target differently from this brief, follow the code and list each difference in the report; where it behaves differently from the specification, stop with Verdict: BLOCKED <reason>.
No real webhook receiver, no real DNS name and no third-party service in any test: receivers are TLS test servers on 127.0.0.1 with a certificate generated in the test for hooks.example.com, reached through a fake resolver of the egress client; nothing in this ticket costs money.
The Standard Webhooks reference library is a test-only dependency pinned in go.mod and go.sum at its newest release; any local copy of a tool or library lives in a throwaway folder from mktemp -d. Never add an allowlist entry or weaken a rule of .gitleaks.toml: every whsec_ value in tests and vectors carries the marker EXAMPLE (CONTRIBUTING.md), so the existing stopword applies; derived secrets appear in the vector file only as derived_hex (Design note 12).
Never move last_delivered_seq or the consumer cursor by hand outside a test database.

Design notes¶

Preconditions, before writing code. Stop with Verdict: BLOCKED <reason> when any of these holds on the base branch: a. docs/spec/actions.yaml lacks the system action webhook_out.transition, with which the automatic subscription transitions of Design note 5 are audited. The core contract has it (05 4.11 row webhook_out.transition: resource app, agent, right system, Owner, Step-up and Permissions no, Audit yes, class none; R-AZ-27, D-W-104), so a base branch without the entry has not taken over that spec state: stop with Verdict: BLOCKED spec state: webhook_out.transition (05 4.11, R-AZ-27) missing in docs/spec/actions.yaml on the base branch. b. The Standard Webhooks reference library for Go cannot be resolved as a Go module or its license is not on quality/licenses/allowed.txt (G10 needs it). Otherwise continue; M1-63, M1-69, M1-70 and M1-71 are used only if merged.
Subscription use cases (apps/app/webhooks.go), one unit of work each, in this order: the generated layer has validated the request schema (unknown fields give 400 validation_failed); authorize(actor, action, Resource{Kind: app, ID: app_id}) with webhook_out.manage (create, update, delete) or webhook_out.read (lists); for operations by subscription or delivery ID, an unknown ID gives 404 not_found before authorize, and authorize runs on the subscription's app, so a non-owner gets the same 404 as for an unknown ID (07 4.3: app is visible to owners only; AZ-P4). Then: a. Create (createWebhookSubscription): lock the app's current installation row FOR SHARE; no installation in active or suspended gives 409 conflict with reason: app_not_installed (the reason M1-44 uses); validate url (Design note 9, 400 webhook_host_not_allowed or 400 validation_failed with pointer /url) and event_types (non-empty, no duplicates, each AppSubscribable and in event_subscriptions of the installation; otherwise 400 validation_failed with pointer /event_types/<index> and errors[].code not_subscribable or not_granted); insert with signing_key_kid = KeyID() of the active webhook_out signer, hmac_enabled = enable_hmac, hmac_generation = 0, state = active, created_by = actor; with HMAC derive the secret (Design note 11), return it once as hmac_secret and call idempotency.MarkOneTimeSecret(ctx), so a replay answers 409 conflict with reason: one_time_secret (06 7, R-API-26). Answer 201 WebhookSubscriptionCreated. b. Update (updateWebhookSubscription, PATCH): lock the subscription row FOR UPDATE; a new url or event_types is validated as in a; state: paused from active or disabled; state: active from paused or disabled (owner resume and re-enable, Design note 5): consecutive_failures = 0, disabled_at = NULL, the head pending delivery (lowest event_seq with status = 'pending') gets next_attempt_at = now, and Schedule(subscription, now); rotate_hmac_secret: true without HMAC gives 400 validation_failed with pointer /rotate_hmac_secret and errors[].code hmac_not_enabled, with HMAC it sets hmac_generation = hmac_generation + 1, hmac_rotated_at = now, returns the new secret once and marks the one-time secret (replay 409 conflict, one_time_secret; this ticket adds the 409 response to the operation). An empty body changes nothing. Answer 200 WebhookSubscriptionCreated. c. Delete (deleteWebhookSubscription): delete the row (deliveries go by ON DELETE CASCADE); 204; a second delete is 404 (no tombstone, 06 7 row "PUT and DELETE"). d. Lists: listWebhookSubscriptions returns every subscription of every installation of the app, ordered by created_at, then id; listWebhookDeliveries pages by (event_seq DESC, id DESC) with the cursor codec of platform/httpserver, filter status, 400 cursor_invalid for a foreign cursor (06 6). app_id, signature_schemes ([v1a] or [v1a, v1]) and event_type are derived (Design note 13). e. Audit: every allowed change writes one entry webhook_out.manage in its transaction through auditing.Recorder.Record (07 4.2: the use case writes allows) with subscription_id, app_id, change (created, updated, paused, resumed, rotated, deleted), host, event_types (list of codes), hmac_enabled, hmac_generation; never the URL path or query and never a secret (SEC-AUD-5). Denials are written by authorize (R-AZ-04).
The consumer webhooks (apps/app/consumer.go), the sync.ConsumerHandler of M2-01, runs in the round's transaction once per event in seq order, writes only apps rows, jobs and the audit entries of its system transitions (webhook_out.transition, R-AZ-27, through auditing.Recorder.Record), calls nothing outside the process and appends no durable event (M2-01 Design notes 2 and 6). For an event ev: a. Skip at once when AppSubscribable(ev.Type) is false. b. Candidates (loaded once per round and memoised under a txn.Key): app subscriptions (app_installation_id IS NOT NULL) in state active or paused whose installation is active or suspended, with ev.Type in event_types and in the installation's event_subscriptions. A disabled subscription gets no rows: events that arrive while it is disabled are not delivered (Ruling for docs/DECISIONS.md, D-M2-26-1; webhooks.md 5 states it). A personal event (ev.AudiencePrincipalID set) goes only to the subscriptions of the installation whose app principal is the audience. Rights are not checked here: they apply at delivery time (04 8.1, B7). c. Per candidate insert webhook_deliveries (id, subscription_id, event_seq, event_id, status 'pending', attempt 0) with ON CONFLICT (subscription_id, event_seq) DO NOTHING. d. Quota (11 6 "Delivery quota"; Ruling for docs/DECISIONS.md, D-M2-26-2): the open count of a subscription (rows with status = 'pending', counted once per round with a bounded count and then tracked in the memo) is compared before and after each insert; when an insert brings an active subscription from below webhook.max_pending_per_subscription to it or above, the subscription becomes paused (row lock FOR UPDATE, system transition of Design note 5, audited). Rows keep being created while paused, so no event is lost; because the rule fires on the crossing, an owner who resumes a subscription with a large backlog is not paused again before it drained below the quota and filled up again. e. Jobs: one Schedule(subscription, now) per active subscription with an active installation that got at least one new row in this round (repeated wake-ups are harmless, Design note 4). f. Lost range (ADR 0051 Consequences, R-EX-07, M2-01 Design note 5): when the consumer webhooks itself fell behind sync.event_retention, the M2-01 round reports LostFrom and LostTo. Those events never became rows, so the head rule of note 5 e cannot see them. internal/sync gains type ConsumerLossHandler interface { HandleLost(ctx context.Context, from, to int64) error }. When PlanRound reports a loss and the registered handler implements this interface, the round calls it once, in the round's transaction, before the first HandleEvent. An error rolls back the round like a handler error. Consumers without the interface (for example push) are unaffected. The webhooks handler implements it as follows: every app subscription in state active or paused whose installation is active or suspended becomes disabled with cause gap (system transition of note 5 b, disabled_at = now), because any of them could have matched a lost event. Their pending rows all belong to events below from, which left the log as well, so they become dead with last_error = event_expired as in note 5 e; the audit entry names from_seq (the lowest event_seq of those rows, else from) and to_seq = to. A re-enable then continues with the events the consumer reads after it, so it never loops. The candidates of note 3 b are loaded after this step.
The attempt (apps/app/deliver.go, job apps.webhook_deliver with args {SubscriptionID}; Ruling for docs/DECISIONS.md, D-M2-26-3). One job kind serves every trigger: a round (note 3 e), the chain after a finished delivery, a scheduled retry, the lease wake-up (note 4 e), an owner resume or re-enable (note 2 b), redeliverWebhook (note 13) and ResumeInstallation (note 14). Several jobs for one subscription are harmless: all serialize on its row lock and only a due head is sent, so at most one attempt per subscription is in flight (strict order, requirement 18). a. Claim, one unit of work: SELECT ... FROM webhook_subscriptions WHERE id = $1 FOR UPDATE; missing or not active: done. Installation not active (for example suspended): done without change. Head = the pending row with the lowest event_seq; none: done. next_attempt_at after now: done (a scheduled retry, the lease wake-up or the attempt in flight owns it). b. Event: sync.LogReader.EventBySeq(head.event_seq); not retained: gap (note 5 e), done. c. Routing: ev.Type no longer in the subscription's event_types or in the installation's event_subscriptions (11 4.2 "Removed ... events ... take effect immediately"): head dead with last_error = not_subscribed, chain, done. d. Rights: authorize(authorize.Filter(ctx), appPrincipal, webhook_out.deliver, Resource{Kind: ev.ResourceKind, ID: ev.ResourceID, SpaceID, ChannelID, ThreadID from ev, DeliveryOf: ev.ReceiveAction, Audience: ev.AudiencePrincipalID}) with appPrincipal = actor.Principal{ID: app principal, Kind: app, InstallationID, GrantedScopes} read in this unit of work (current grants, scopes and installation). Any outcome other than allow: head dead with last_error = not_authorized, attempt unchanged, no request, consecutive_failures unchanged, chain, done (requirement 20, AZ-M49). An error (not a decision) returns the error, and River retries the job. If M1-63 merged a mapping from a stored event to the resource of its receive action, use it instead of building the descriptor here. e. Request: body = json.Marshal(ev.Envelope(sync.DeliveryFields{SyncEpoch: instance.State.Load(ctx).SyncEpoch})); ts = clock.Now() in Unix seconds; signatures of note 11. Lease: attempt = attempt + 1, next_attempt_at = now + webhook.delivery_timeout, and in the same unit of work Schedule(subscription, next_attempt_at) as the lease wake-up, so an attempt whose send or record step ends with an error, a cancellation or a crash is sent again once the lease has run out; commit. Keep body and header in memory only. f. Send through WebhookSender outside any transaction (02 8.5). g. Record, one unit of work: lock the subscription row, reload the head; if it is no longer pending with the claimed attempt (deleted, redelivered, paused and changed), discard the result. Otherwise apply the outcome of note 6 with the timing of note 7, set response_code when a response arrived, and chain: Schedule(subscription, now) after a finished delivery while the subscription stays active; Schedule(subscription, next_attempt_at) after a retryable outcome; nothing when the subscription became disabled or is no longer active. h. On a 2xx: head succeeded, delivered_at = now, next_attempt_at = NULL, last_error = NULL; subscription last_delivered_seq = GREATEST(last_delivered_seq, event_seq), consecutive_failures = 0, signing_key_kid = the kid of the active key that signed (Ruling for docs/DECISIONS.md, D-M2-26-4: the field names the key that signed the last accepted delivery, at creation the active key). Locks: the subscription row first, then its delivery rows; a round locks a subscription row only for the quota transition and never waits for the consumer cursor (M2-01), so no cycle exists. The job's own failures (database, context cancelled) return the error and River retries with its defaults (M1-14 rules 15 and 16); a retry that still finds the lease ahead ends, and the lease wake-up of e sends the head again; the attempt is idempotent through attempt and the lease. A crash after the receiver answered and before g leads to a second send with the same webhook-id (at least once, webhooks.md 5 point 1; the receiver deduplicates).
Subscription states (domain/subscription.go; 03 state; webhooks.md 4 and 5; 11 6): a. Owner: active -> paused, paused -> active, disabled -> active (re-enable), disabled -> paused; resume and re-enable reset consecutive_failures and make the head due now (note 2 b). b. System, each audited with the system action webhook_out.transition (05 4.11, R-AZ-27; actor none, resource app, details subscription_id, from_state, to_state, cause, and from_seq, to_seq for a gap), in the transaction of the change, and logged at warn level: 410 -> disabled (cause gone); consecutive_failures reaches webhook.max_consecutive_failures -> disabled (cause max_consecutive_failures; the head stays pending and goes first after a re-enable); a delivery dead after webhook.retry_max_age -> disabled (cause retry_max_age); a gap -> disabled (cause gap); the quota of note 3 d -> paused (cause max_pending). disabled_at = now on every change to disabled. "the owner is notified" (11 6) is this audit entry plus the state in listWebhookSubscriptions; the product sends no mail (webhooks.md 4 point 2) and 05 has no event type for subscriptions. c. uninstallApp sets every subscription of the installation disabled in its transaction (11 4.3; covered by the audit entry of app.uninstall). d. After a re-enable delivery continues with the remaining pending rows in seq order and then with events committed after it (webhooks.md 5 point 3 "next event"). e. Gap (Ruling for docs/DECISIONS.md, D-M2-26-5; 11 6, R-EX-07): detected at the head when its event is no longer in the log (EventBySeq false). In one unit of work: subscription disabled (cause gap), every pending row of the subscription with event_seq below MinRetainedSeq() becomes dead with last_error = event_expired, the audit entry names the first and last lost seq. A re-enable then continues with the first retained pending row, so it never loops. Among the rows that exist, this detects exactly the deliveries that were due, without false positives for subscriptions that match rarely. A loss inside the consumer itself is detected by note 3 f.
Outcomes and statuses (domain/delivery.go Classify; Ruling for docs/DECISIONS.md, D-M2-26-6). Statuses of 03: pending (waiting for its first or next attempt, also after a failed attempt; next_attempt_at set), succeeded (2xx), failed (finished without success because the receiver answered 410; terminal, the owner may redeliver), dead (finished: webhook.retry_max_age passed after a failure, not_authorized, not_subscribed, event_expired). last_error is a code from the closed list timeout, connection_failed, tls_failed, address_blocked, host_not_allowed, http_status, throttled, gone, not_authorized, not_subscribed, event_expired (never a response body, 03; webhooks.md 4 point 5). Mapping: 2xx -> success; 410 -> failed, gone, subscription disabled; 429, 502, 503, 504 -> retry, throttled, not counted; any other status including 3xx (the egress client returns it unchanged with RedirectNever) -> retry, http_status, counted; egress.ErrBlocked with reason address_blocked -> retry, address_blocked, counted; reasons host_not_allowed, scheme_not_allowed, url_not_allowed -> retry, host_not_allowed, counted; deadline -> timeout; TLS handshake or certificate error -> tls_failed; other transport errors -> connection_failed; all counted. "Counted" means consecutive_failures + 1, compared with webhook.max_consecutive_failures (webhooks.md 4 table separates "Throttle" from "Failure"; only "failures" count, point 2). Retention (M7-05) may delete only succeeded, failed and dead rows older than retention.webhook_deliveries, never pending ones: state this in the apps README and in webhooks.md 4.
Timing (Ruling for docs/DECISIONS.md, D-M2-26-7). After the n-th counted or throttled failure in a row of this delivery (n = attempt), the backoff is a uniform random duration in [webhook.retry_backoff_base, min(webhook.retry_backoff_cap, webhook.retry_backoff_base times two to the power n-1)]: exponential with full jitter between base and cap (webhooks.md 4 point 1), the random source injected for tests. For a throttle with a valid Retry-After (delta-seconds or HTTP-date of RFC 9110 10.2.3; a past date counts as zero; anything else is ignored) the delay is the larger of backoff and Retry-After (webhooks.md 4 table). The retry deadline is created_at + webhook.retry_max_age: a retry is never scheduled after it (a later time is cut to the deadline), and a counted or throttled failure at or after it makes the delivery dead and the subscription disabled (webhooks.md 5 point 3). The deadline is checked only after a failed attempt, never before sending, so a head that waited in a paused or disabled subscription still gets its attempt after the resume.
webhook_out.deliver in authz (07 4.2 "Requirements that need facts of a later milestone ... The ticket that delivers the facts replaces the entry"). Requirement atom Delivery in domain/requirements.go, evaluated in decide.go, in this order: principal kind app (an agent gives forbidden subject_kind_not_allowed until the agent transport, Out of scope; any other kind the same); the liveness check of M1-28 has run (app_not_installed for an installation that is not active); scope events:subscribe in GrantedScopes, else forbidden scope_missing (05 2.6 "Event subscriptions"); with Audience set: Audience equal to the principal, else not_found not_self, then allow (receive action self, and for apps agent.events.consume is replaced by events:subscribe, 03 event_log.receive_action); otherwise the full decision of DeliveryOf on the same resource (visibility, requirement, the app scope of DeliveryOf from domain/scopes.go, for example messages:read for message.read, none for instance.read, status.read, command.read, agent.read), whose outcome is the result. DeliveryOf empty or itself webhook_out.deliver is an internal error. Only in filter mode from the job: no denial is written and no decision is counted (07 4.2, AZ-P13). writes(webhook_out.deliver) stays false (M1-28). Owners have no special rule here: an app is never owner. The rights version does not change.
Targets (domain/subscription.go ValidateTarget; Ruling for docs/DECISIONS.md, D-M2-26-8): entries of webhook.allowed_hosts have the form host[:port] (host lower case, no scheme, no path; without port only the default port of https); a URL is accepted when it parses, its scheme is https, it has no user info and no fragment, and its canonical host (lower case, no trailing dot) and port match an entry; otherwise 400 webhook_host_not_allowed for a host or port not in the list and 400 validation_failed (pointer /url) for every other defect. The same list is the HostPolicy of the webhook egress client, so a list changed after a restart blocks old subscriptions with host_not_allowed at delivery (counted). Private addresses need an entry in egress.allowed_hosts with allow_private (operator; webhooks.md 6 point 2); a subscription never opens http. This rule replaces the host-without-port comparison of M2-24 Design note 5: domain.ValidateManifest checks delivery.events_url and delivery.interactions_url with ValidateTarget and the same parsed []AllowedHost (host or port not listed: item code webhook_host_not_allowed; other defects keep M2-24's codes). Manifest addresses, subscription URLs and the egress HostPolicy then follow one host[:port] rule, and the install check that Design note 14 relies on is this check.
Subscribable types (Ruling for docs/DECISIONS.md, D-M2-26-9): AppSubscribable is the set of keys under webhooks: of docs/api/openapi.yaml minus the event types with receivers: agent in docs/spec/actions.yaml (the "agent subscription only" rows of webhooks.md 7). Keep it as a hand-written table in domain with a test that reads both files from the repository and fails on any difference.
Signatures (Rulings for docs/DECISIONS.md, D-M2-26-10 and D-M2-26-11): a. v1a: one signature per signer of instance.Keys.Signers(ctx, webhook_out) (active first, then every retiring key whose not_after lies ahead; webhooks.md 2.1 point 2: "the server signs with both until the old one reaches not_after"). With the promotion rule of M1-32 (not_after = time of promotion) this is the active key alone; the rule stays general so a later change of the window needs no change here. b. v1 when hmac_enabled: key of hmac_generation, plus the key of hmac_generation - 1 while hmac_rotated_at + webhook.secret_rotation_overlap lies ahead; never older generations. c. Header: entries separated by one space, all v1a entries first in signer order, then the v1 entries newest generation first; each entry <scheme>,<standard base64 with padding> (webhooks.md 2). d. HMAC derivation: HMAC-SHA256(master, "webhook-v1:" + subscription_id + ":" + generation) with the subscription ID in canonical lower-case hyphenated form and the generation in decimal ASCII; the master key is the hex-decoded value of webhook.hmac_master_key_ref (M1-09 D5). The derived bytes are the HMAC key; whsec_ + standard base64 of them is the secret shown once, the form the reference libraries decode. e. whpk_ + standard base64 of the DER-encoded SubjectPublicKeyInfo (RFC 8410) of the Ed25519 key, the form of the example under getWebhookKeys (Ruling D-M2-26-11; the openapi description of WebhookKey.public_key names it). f. getWebhookKeys (instance/adapters/httpapi): PublicKeys(ctx, webhook_out) ordered by state (active, next, retiring) and kid; each item {kid, public_key, state, not_after} with not_after only when set (no null); Cache-Control and Access-Control-Allow-Origin: * exactly as getJwks sets them (M1-32: a next key is published agent.jwks_max_age before it signs; SEC-CORS-2). g. Secrets never leave memory except in the one response: no log, no audit, no error text, no row (including idempotency_keys), no span (SEC-SEC-3, R-API-26).
Test vectors (Ruling for docs/DECISIONS.md, D-M2-26-12): docs/api/webhook-test-vectors.json, the contract file for every verifier (M2-29, M2-30): {"version": 1, "derivation": [...], "signatures": [...]}; each derivation case {master_hex, subscription_id, generation, derived_hex}, with derived_hex the lowercase hex of the 32 derived bytes. The secret shown to the owner is whsec_ + standard base64 of those bytes (note 11 d) and is not stored in the file, because a derived value cannot carry the EXAMPLE marker. master_hex values are low-entropy fixed byte sequences (for example the bytes 00 to 1f), so no gitleaks rule matches them. The hmac_secrets of the signature cases are chosen whsec_EXAMPLE... values (valid standard base64), not derived ones; each signature case {name, webhook_id, timestamp, body, ed25519_seed_base64, public_key, hmac_secrets, signature_header} with at least: one key and one secret; two v1a signers; two HMAC generations (overlap); a body with non-ASCII characters and escapes; a body with truncated: true. Values are generated once by a test helper in this ticket from fixed seeds and checked in; every whsec_ value contains EXAMPLE (gitleaks stopword). webhooks.md 2.2 documents the format and the field names.
Deliveries API (Rulings for docs/DECISIONS.md, D-M2-26-13 and D-M2-26-14): a. event_type is not a column of 03 webhook_deliveries; the list reads it with EventBySeq while the event is retained and omits it afterwards. WebhookDelivery.event_type leaves required in openapi.yaml with the description "absent once the event left the event log (sync.event_retention)" (the core contract wins over the API document; spec gap reported). b. redeliverWebhook (webhook_out.redeliver, resource webhook_delivery): a pending row gets next_attempt_at = now and a job (retry now); a succeeded, failed or dead row becomes pending again with next_attempt_at = now, last_error and response_code cleared, attempt kept and created_at = now(), so the retry window of webhook.retry_max_age starts again: a redelivery is a new delivery of the same event with the same webhook-id (webhooks.md 4 point 4). Being the lowest pending seq, it goes out before the subscription's later pending rows (strict order). An event that is no longer in the log gives 409 conflict with reason: event_expired (this ticket adds the 409 response to the operation). A paused or disabled subscription keeps the row waiting. Answer 202 WebhookDelivery; audit webhook_out.redeliver with delivery_id, subscription_id, event_seq, previous_status; Idempotency-Key through M1-20.
Install, uninstall, suspend (Ruling for docs/DECISIONS.md, D-M2-26-15; 11 4.1, 4.2, 4.3): the install use case of M2-24 creates, when the manifest has delivery.mode = webhook, one subscription with url = delivery.events_url, event_types = the installation's event_subscriptions, no HMAC and created_by = the installing owner, in the install transaction, after authorize(webhook_out.manage) and with its own audit entry (change = created, same correlation_id); the host check of M2-24 (11 3.2 point 3) already ran. Upgrades and scope changes never change existing subscriptions: removed event types stop at delivery (note 4 c), new ones need an owner updateWebhookSubscription. Uninstall disables (note 5 c). A suspended installation keeps getting rows and sends nothing (note 4 a); ResumeInstallation makes the head of each active subscription of the installation due now and schedules a job, in the caller's unit of work.
Configuration (Ruling for docs/DECISIONS.md, D-M2-26-16): internal/wire/configkeys_webhook.go registers webhook.max_consecutive_failures (int, at least 1), webhook.retry_backoff_base, webhook.retry_backoff_cap (durations above zero) with chapter03, and webhook.max_pending_per_subscription (int, at least 1), webhook.retry_max_age, webhook.secret_rotation_overlap (durations above zero) and the new webhook.hmac_master_key_ref (the kind the merged schema uses for *_ref keys such as database.password_ref, default file:<secrets.dir>/webhook_hmac_master_key) with chapter11; keep webhook.delivery_timeout as M1-32 registered it; keep webhook.allowed_hosts as M2-24 registered it in configkeys_apps.go (one registration; Design note 9 defines its entry form); webhook.receiver_tolerance is a receiver value and gets no server key (D-W-23). Cross checks in configchecks.go, added to 02 13 item 4 "Cross conditions today": webhook_backoff_order (webhook.retry_backoff_base at most webhook.retry_backoff_cap) and webhook_retry_within_retention (webhook.retry_max_age strictly below sync.event_retention, because a retry needs its event in the log); messages name the keys, never a value. New row in 11 16: webhook.hmac_master_key_ref | Server | file:<secrets.dir>/webhook_hmac_master_key | reference to the master key from which the HMAC secrets of the subscriptions are derived ([webhooks.md](../api/webhooks.md), section 2.1); row in configuration.md 5: webhook_hmac_master_key | generated | server | webhook.hmac_master_key_ref: master key of the derived webhook secrets. Changing the master key changes every derived secret; the apps README says so and points to rotate_hmac_secret.
Telemetry (Ruling for docs/DECISIONS.md, D-M2-26-17): span webhook.deliver around the HTTP exchange with server.address (host only), http.response.status_code when a response arrived, bothy.outcome from the closed list succeeded, throttled, gone, failed, blocked. Logs with constant English messages, attributes only from signals.md 5.3: webhook delivery failed (info) and webhook subscription state changed (warn) with bothy.webhook_subscription.id (new attribute, UUID, never on metrics), bothy.event.seq, bothy.event.type, bothy.reason (the last_error code or the cause of note 5 b) and server.address. Add the attribute to signals.md 5.3, otel/attr and the collector allowlist in the same change. No metric is added; bothy.egress.blocked of M1-22 counts refused attempts.
Threat model 5.11, rows to add (English, like the table): S | Forged or replayed webhook at the receiver | v1a always, v1 per subscription, webhook-timestamp and webhook-id for freshness and duplicates, test vectors (G10) | T-PL-01; I | App receives events outside its grants or scopes | webhook_out.deliver before every attempt with current permissions, dead without a request | T-PL-01; I | Webhook to a private target (DNS rebinding) | host list webhook.allowed_hosts, egress client with an address check at connection setup, no redirects | T-NW-01; I | HMAC secret from the database or the idempotency store | derived, never stored, one-time response | T-TK-02; D | Slow receiver congests deliveries | order per subscription, queue delivery, delivery quota, disabling | T-DS-01.
Rulings for docs/DECISIONS.md section 10, one row each with date, decision, reason and place: D-M2-26-1 rows only for active and paused subscriptions, none while disabled (note 3 b); D-M2-26-2 quota as pause trigger on the crossing (note 3 d); D-M2-26-3 one job kind per subscription with chain, lease and lease wake-up (note 4); D-M2-26-4 signing_key_kid names the key of the last accepted delivery (note 4 h); D-M2-26-5 gap detected at the head and from the lost range of the round, all expired pending rows dead (notes 3 f and 5 e); D-M2-26-6 statuses, last_error codes, throttles not counted (note 6); D-M2-26-7 backoff, Retry-After and deadline (note 7); D-M2-26-8 target rules and the host[:port] form, replacing the host-without-port comparison of M2-24 Design note 5 (note 9); D-M2-26-9 subscribable set (note 10); D-M2-26-10 signing set, header order, HMAC overlap and derivation encoding (note 11 a to d); D-M2-26-11 whpk_ as DER SubjectPublicKeyInfo (note 11 e); D-M2-26-12 test vector file, derived secrets only as derived_hex (note 12); D-M2-26-13 event_type derived and optional (note 13 a); D-M2-26-14 redelivery semantics and event_expired (note 13 b); D-M2-26-15 install, uninstall, suspend (note 14); D-M2-26-16 configuration keys, master key secret and cross checks (note 15); D-M2-26-17 telemetry (note 16); D-M2-26-18 validation codes not_subscribable, not_granted, hmac_not_enabled and the reasons app_not_installed, one_time_secret, event_expired (notes 2 and 13); D-M2-26-19 agent subscriptions not delivered and subject_kind_not_allowed for agents until the owning ticket (Out of scope); D-M2-26-20 deliveries listed newest first (note 2 d).
How the M2 section applies. "Goal" (P-18): the delivery uses the same envelope as gateway and REST. "Scope": webhooks of 06 and webhooks.md, ADR 0016 and 0051. "Exit criteria" 3 and 4: AZ-M49 and G10 run in make check with the integration tests. "Test strategy": the chaos item "duplicate delivery" is the crash test of note 4 (same webhook-id). "Red team": "webhooks to private targets" and "exceeding the scopes of an app" are the tests TestDeliveryPrivateAddressRefused and TestDeliveryScopeRemovedIsDead; webhook targets are a boundary in the threat model (note 17). "Risks": RK-04 and RK-20 are reduced by notes 9, 11 and 12; name them in the report.

Tests (write them first)¶

Unit:

internal/apps/domain: TestClassify (table: 200, 204, 299 success; 410 gone; 429, 502, 503, 504 throttled; 300, 302, 307, 400, 401, 404, 500, 501 counted http_status; each egress reason, a deadline, a TLS error and a reset connection to their codes).
TestBackoff and FuzzBackoff (native Go fuzz test whose seeds run in go test): every result lies in [base, cap]; the first retry waits exactly base; the upper bound doubles up to cap; the injected random source drives the jitter.
TestParseRetryAfter and FuzzParseRetryAfter: delta-seconds, HTTP-date, a past date, negative, empty, garbage; never panics.
TestNextAttempt: backoff versus Retry-After, the cut at the deadline, dead at and after the deadline.
TestSubscriptionTransitions: every owner and system transition of Design note 5 allowed, every other rejected (for example quota on a paused or disabled subscription changes nothing).
TestValidateTarget and FuzzValidateTarget: https only, user info and fragment rejected, host not listed gives webhook_host_not_allowed, port rules, upper case and trailing dot normalised, IP literal not listed rejected; no input panics and an accepted URL always has scheme https.
TestValidateEventTypes: empty, duplicates, not_subscribable, not_granted.
TestAppSubscribableMatchesContract: the set equals the webhooks: keys of openapi.yaml minus receivers: agent of actions.yaml (Design note 10).
TestSignatureHeaderFormat, TestHMACOverlap (generation and the previous one only inside webhook.secret_rotation_overlap), TestDeriveHMACKeyStable (a derivation vector of the file; another subscription or generation gives another key).
G10, part 1 (internal/apps/domain): TestWebhookTestVectors: for each case of docs/api/webhook-test-vectors.json the domain functions reproduce signature_header byte for byte from seed, secrets, ID, timestamp and body; each v1a signature verifies with crypto/ed25519 against the key decoded from public_key; each v1 signature verifies with the Standard Webhooks reference library; a body changed by one byte fails both; each derivation case reproduces derived_hex with DeriveHMACKey, and ParseSecret(FormatSecret(key)) returns the same bytes.
internal/sync/app: TestRoundCallsHandleLost (a handler implementing ConsumerLossHandler gets the lost range once, before the first event; an error rolls back the round, so neither the cursor nor a job moves; a handler without the interface runs as before).
internal/instance: TestFormatWebhookPublicKey (round trip through x509.ParsePKIXPublicKey, prefix whpk_), TestSignersOrder (active first, retiring with not_after ahead included, past not_after, next and retired excluded), TestPublicKeysWebhookOut.
internal/authz: TestWebhookOutDeliverRequirement (table: app with grant and scopes events:subscribe and messages:read on message.created gives allow; no grant on the channel gives not_found; without messages:read gives forbidden scope_missing; without events:subscribe gives forbidden scope_missing; personal event for another principal gives not_found; personal event for the app itself gives allow; instance.updated needs only events:subscribe; agent principal gives forbidden subject_kind_not_allowed; empty DeliveryOf is an internal error); TestDeferredActionsGolden without webhook_out.deliver; TestWebhookDeliverFilterWritesNoDenial (AZ-P13).
internal/wire: TestConfigCheckWebhookBackoffOrder, TestConfigCheckWebhookRetryWithinRetention (equal fails, greater fails, smaller passes; messages name both keys and no value); TestConfigSchemaBuilds with the new keys; TestWebhookConsumerRegisteredInWorkerRoles; TestWebhookJobRegistration (kind, queue delivery, args pass jobs.ValidateArgsType).
internal/platform/secrets/stack: the catalogue test of M1-09 passes with the new row in configuration.md 5.

Integration (PostgreSQL 18 via pgtest, roles from migrate --bootstrap, runtime as bothy_app; the world of 07 10 and 11.1 from BuildMatrixWorld of M1-71 if merged, otherwise from the test helpers of the modules: spaces Support (channels Allgemein and Eskalation) and Marketing, people Alex and Sam, app Tracker installed with scopes events:subscribe, messages:read, messages:write and event_subscriptions = [message.created] and grant G-6 (write, Support/Eskalation); receivers are TLS test servers of the Boundaries; a fake clock where noted; jobs run through a jobs.Runtime in role worker, or in role api where a test needs rows without running jobs):

TestWebhookTablesMatchDataModel: columns, types, nullability, defaults, checks (both exclusivity checks with SQLSTATE 23514), foreign keys with their actions, unique key, indexes; bothy_app privileges; TestPrivilegesAfterAllMigrations of M1-12 still passes.
TestCreateWebhookSubscription: owner with fresh step-up gets 201 with signature_schemes = [v1a], signing_key_kid of the active key, no hmac_secret; with enable_hmac the secret appears once with prefix whsec_ and signature_schemes = [v1a, v1]; replay with the same Idempotency-Key answers 409 conflict with reason: one_time_secret and Idempotent-Replay: true, and no row of idempotency_keys contains any part of the secret; without HMAC the replay returns the stored 201; host not listed 400 webhook_host_not_allowed; http URL 400 validation_failed with pointer /url; type not granted 400 validation_failed with errors[].code = not_granted; uninstalled app 409 conflict with reason: app_not_installed; Alex 404 with the same status, body and headers as an unknown app ID; owner with stale step-up 401 step_up_required; owner through a personal token 404; one audit entry with host and without path.
TestUpdateWebhookSubscription: pause, resume, re-enable from disabled (failures reset, head due, a job inserted); rotate_hmac_secret raises the generation, returns the new secret once, and deliveries inside webhook.secret_rotation_overlap carry both generations, after it only the new one; rotation without HMAC 400 validation_failed with hmac_not_enabled; replay of a rotation 409 conflict one_time_secret; a new host is checked.
TestDeleteWebhookSubscription: 204, deliveries gone, second delete 404, audit entry.
TestListWebhookSubscriptionsAndDeliveries: order, app_id, pagination with next_cursor, status filter, 400 cursor_invalid; a delivery whose event left the log has no event_type and validates against WebhookDelivery.
TestGetWebhookKeys: anonymous 200; only webhook_out keys in next, active, retiring; public_key with prefix whpk_ decoding to the key; Cache-Control and Access-Control-Allow-Origin as getJwks; the body validates against WebhookKeys; getJwks contains no webhook_out key.
TestWebhookRoundCreatesDeliveries: one row per active and paused candidate, none for disabled subscriptions or uninstalled installations, none for types outside event_types or event_subscriptions; a personal event only for the audience app; one job per active subscription per round; a second round over the same events changes nothing.
G10, part 2: TestWebhookDeliveryG10: Alex posts in Support/Eskalation; the receiver gets one POST with Content-Type: application/json, User-Agent chat-server-webhooks/<version>, webhook-id = event ID, webhook-timestamp of the attempt; the body is the envelope of the stored event with sync_epoch = instance.sync_epoch and contains no null; v1 verifies with the reference library and the whsec_ secret from the create response; v1a verifies with crypto/ed25519 and the key from getWebhookKeys; the row is succeeded with response_code and delivered_at; last_delivered_seq equals the event's seq. Use the real clock or a fake clock set to now, because the reference library checks the timestamp tolerance.
TestAZM49: Tracker's subscription on message.created; Alex posts in Support/Allgemein and then in Support/Eskalation; the receiver gets exactly the Eskalation event; the Allgemein row is dead with last_error = not_authorized, attempt = 0, no response_code; consecutive_failures is zero; no audit denial is written.
TestDeliveryRevokedRightIsDeadWithoutRequest (plan: "a revocation of permissions yields dead without a request"; ADR 0051 "Confirmation"): the round creates the row with jobs held back (role api), the owner revokes G-6, then the job runs: no request reaches the receiver, the row is dead with not_authorized, the next event of the subscription goes out after a new grant. Variant TestDeliveryScopeRemovedIsDead: setAppScopes removes messages:read between round and job, same result (Red Team "exceeding the scopes of an app").
TestDeliveryStrictOrder: three events; the receiver answers 500 twice to the first; the receiver log shows the second and third only after the first's 2xx, in seq order.
TestDelivery410Disables (plan: "410 disables"): row failed with response_code 410 and last_error = gone; subscription disabled with disabled_at; one audit entry of webhook_out.transition with cause gone; no further request; after a re-enable the next event goes out.
TestDeliveryRetryAfter (plan: "429 and 503 respect Retry-After"): 429 with Retry-After in seconds and 503 with an HTTP-date, each larger than the backoff: next_attempt_at is not earlier than the Retry-After time and the job is scheduled at it (fake clock); consecutive_failures stays unchanged; the same for 502 and 504; a Retry-After beyond the deadline is cut to it.
TestDeliveryBackoffAndConsecutiveFailures: 500 answers: every next_attempt_at lies in the backoff interval of Design note 7; at webhook.max_consecutive_failures (small value from the test configuration) the subscription is disabled, audited with cause max_consecutive_failures, the head stays pending; a re-enable sends the head first.
TestDeliveryDeadAfterRetryMaxAge: with the fake clock past created_at + webhook.retry_max_age, a failure makes the row dead and the subscription disabled; after a re-enable the following event goes out.
TestDeliveryPrivateAddressRefused (plan: "a private target address rejected"; Red Team "webhooks to private targets"): the fake resolver answers hooks.example.com with a loopback address and with a private address of RFC 1918, without an allow_private entry: no connection reaches the receiver, the row stays pending with last_error = address_blocked and a counted failure, bothy.egress.blocked records address_blocked through the fake metrics of the egress client, and the log contains host and reason only. With the entry hooks.example.com:<port>;allow_private in egress.allowed_hosts the same delivery succeeds. A receiver answering 302 to another allowed host: the other host gets no request, last_error = http_status.
TestDeliveryGapDisables: a paused subscription with pending rows; the fake clock and the partition maintenance of M1-16 drop the partition of the first event; after resume the subscription is disabled with cause gap, every expired pending row is dead with event_expired, the audit entry names the lost range; a second re-enable delivers the first retained event.
TestConsumerLossDisablesSubscriptions: one active and one paused subscription (the paused one with pending rows), and the consumer cursor behind a partition dropped through the fake clock and the partition maintenance of M1-16. After the round both subscriptions are disabled with cause gap, their pending rows are dead with event_expired, the audit entries name the lost range, and no row or job exists for the retained events of that round. A re-enable delivers the next new event in order and does not disable again.
TestDeliveryQuotaPauses: webhook.max_pending_per_subscription small from the test configuration, the receiver blocks: the crossing round pauses the subscription with an audit entry of cause max_pending; rows keep being created; after resume no immediate new pause; the backlog drains in order.
TestDeliverySuspendedInstallation: installation suspended: rows are created, no request is sent, rows stay pending; ResumeInstallation delivers them in order.
TestInstallCreatesManifestSubscription and TestUninstallDisablesSubscriptions (11 4.1, 4.3).
TestRedeliverWebhook: a dead row becomes pending with created_at moved, goes out with the same webhook-id, ends succeeded; a pending row in backoff goes out at once; an event no longer in the log gives 409 conflict with reason: event_expired; Alex gets 404; the audit entry exists; the replay returns the stored 202.
TestDeliveryConcurrentJobs: two jobs of one subscription at once lead to exactly one request.
TestDeliveryCrashAfterSend (plan "Test strategy": duplicate delivery): with a short webhook.delivery_timeout from the test configuration, the record step fails once after a 2xx (injected store error); River's retry finds the lease and ends; after the lease the lease wake-up job sends again; the receiver sees the same webhook-id twice; the row ends succeeded once.
TestDeliveryDeletedDuringAttempt: a subscription deleted while the receiver holds the request: the result is discarded, no error.
TestWebhookSecretsNeverLeak (SEC-SEC-3, SEC-T10): the whsec_ value, the master key bytes, the URL path and a message text sent through a delivery appear in no row of webhook_subscriptions, webhook_deliveries, idempotency_keys and audit_events (row_to_json), in no captured log record, span attribute or error string. If M1-69's canary scenario exists, extend it with a webhook delivery and the whsec_ value instead of a separate test.
Route registration: AZ-P1 (M1-17, M1-25) passes with the seven enabled operations; VerifyRegistrations accepts them now that webhook_out.deliver is no longer deferred.

There is no end-to-end test, no platform run and no contract test beyond G10: the compose smoke test of M1-73 keeps passing. Plan row test IDs: G10 (Tests "G10, part 1" and "G10, part 2"), AZ-M49 (TestAZM49); the plan row's other items map to TestDelivery410Disables, TestDeliveryRetryAfter, TestDeliveryPrivateAddressRefused and TestDeliveryRevokedRightIsDeadWithoutRequest.

Commands that must pass, from the repository root:

cd services/server && go test -race -count=1 ./internal/apps/... ./internal/authz/... ./internal/instance/... ./internal/sync/... ./internal/kernel/... ./internal/platform/... ./internal/wire/... ./migrations/... ./api/...
make migrations-lock
make migrations-lint
make codegen-check
make arch
make register-check
make api-lint
make api-checks
make api-tables
make config-reference
make config-doc-check
make adr-index
make gitleaks
make licenses
make lint
make docs
make check

After committing, git diff --exit-code docs/operations/configuration.md docs/adr/README.md docs/spec/06-api.md docs/api/webhooks.md shows nothing after make config-reference, make adr-index and make api-tables.

Acceptance criteria¶

Definition of done¶

Code and tests as specified; doc comment on every public symbol (purpose, parameters, return value, errors, example where useful); comments explain why.
READMEs of apps (port Webhooks, tables, consumer, job and its triggers, states and statuses, last_error codes, retention rule of Design note 6, master key, test vectors), sync (LogReader, SyncEpoch, ConsumerLossHandler), instance (Signers, PublicKeys, getWebhookKeys) and authz (requirement of webhook_out.deliver, deferred list) updated.
Rulings of Design note 18 recorded in docs/DECISIONS.md (one row each with reason).
services/server/CHANGELOG.md entry under 'Unreleased'.
ADR 0016 and ADR 0051 "Confirmation" name this ticket's tests without changing the decisions; a new ADR only if you take an architecture decision beyond this brief (MADR 4 as in docs/adr/0000-template-madr-4.md, headings and text in English, next free number, make adr-index).
make check green.
An independent code review over the diff, then the red team gate (docs/quality/red-team-gate.md, verdict JA or NEIN), with attention to the M2 "Red team" items "webhooks to private targets" and "exceeding the scopes of an app"; on NEIN fix with a test per finding and have the delta checked again. Record both verdicts.
Land the change as AGENTS.md ("How work lands on main") describes: a pull request from this branch, squash-merged only when review, red team and make check, ci-ok and the review threads are green; then git worktree remove.

Report¶

Write to {{REPORT_PATH}}:

First line exactly Verdict: DONE or Verdict: BLOCKED <reason>.
Commits (SHA and subject), changed files with line counts, test results with counts (the red run before the change and the green run after), the summary of make check.
Review and red team verdicts and the findings fixed.
What is not proven (for example a receiver that verifies v1a with a library other than the standard library, behaviour under real DNS).
Follow-ups that could not be done in this scope, each with its reason (agent subscriptions and their two operations, suspendApp and resumeApp, the retention rule for M7-05), every difference between the M1 and M2 names in this brief and the merged code (D-W-100), and spec gaps found.

Then write {{DONE_PATH}} with exit=0 sha=<sha> or exit=1 <reason>.