M2-28: Cost reconciliation as its own app¶

Status: ready | Depends on: M2-22, M2-24 | Milestone: M2

Goal¶

After this ticket the estimated costs of agent runs can be replaced by the amounts of the provider's invoice data without the provider's admin key ever reaching the server. A separate app in plugins/cost-reconciler (its own process, default off, no inbound port) reads the admin key from its own secret provider, fetches daily cost buckets per provider workspace, distributes each day's amount over the estimates of the agents mapped to that workspace (in proportion to the estimates, the rounding remainder to the largest one, anything without a run reported as not assigned, BR-10) and submits the result with an app token that carries only the scope costs:reconcile (submitCostReconciliation, action cost.reconcile.submit, no step-up). An owner can do the same by hand with step-up (reconcileCosts, action cost.reconcile); app, agent and personal tokens are refused there (R-AZ-03). Both ways store new reconciled entries next to the estimates, never change or delete an existing entry, and apply them through the same job billing.reconcile, which brings runs and agent_sessions (amounts and the estimate flags) up to date. Why: the estimate from tokens and a price table misses surcharges, discounts and price tiers (10 12.1 to 12.3, RK-16), and an admin key of the provider organisation can manage members and keys, so a break-in into the chat server must not reach it (10 12.4, ADR 0055, RK-17, baseline B10 and I5).

Read first¶

docs/spec/10-backend-policy.md: 1 "Purpose" (the server knows no credentials), 3 "Invariants" (I5), 12.1 "Recording" (one entry per model at the end of a run, all entries estimates until a reconciliation replaces them, no currency conversion), 12.2 "Display" (amounts only for organization, the label "Estimate"), 12.3 "Price table", 12.4 "Reconciliation with billing" (all five steps and the paragraph "Authorization"), 14 "Audit trail" (row "Cost reconciliation"), 15 "Tests" (BR-10, BR-13), 16 (E-BR-07), 17 "Configuration values of this chapter" (reconciler.min_request_interval, reconciler.lookback, reconciler.workspace_map).
docs/adr/0055-cost-reconciliation-as-its-own-app-admin-key-never-in-the-server.md (all, especially "Decision Outcome" points 1 to 5 and "Confirmation"); ADR 0022 (extensions outside the server), 0024 (backend policy), 0011 (secret provider), 0013 (spec first), 0015 (errors, idempotency, pagination), 0017 (one authorize function), 0020 (step-up for admin actions), 0021 (audit), 0049 (action catalogue), 0053 (no JSON null), 0044 (places of the working name).
docs/spec/05-actions.md: rule 1.1, rule 1.2 (its exception for the estimate list, R-AZ-28), rule 1.9 (maintenance layer, effect write), 2.3 "Effective level" point 1 (owner rights only in a session; a personal token of an owner has none), 2.4 'Columns "Owner", "Step-up", "Audit", "Permissions"', 2.5 "Decision", 2.6 "Scopes for apps" (row cost.reconcile.read, cost.reconcile.submit, and the sentence that agents lack costs:reconcile), 2.7 "Effect (effect)", 3 "Rate-limit classes" (rl.admin, rl.read), 4.12 rows cost.read.thread, cost.read.aggregate, cost.reconcile, cost.reconcile.read, cost.reconcile.submit; R-AZ-03, R-AZ-04, R-AZ-28 (the read action for the estimates, D-W-104).
docs/spec/07-permissions.md: 2.10 "Apps" (level plus scope, scope_missing gives 403), 4.1 (reasons owner_only, scope_missing, subject_kind_not_allowed, step_up_required, step_up_not_possible, app_not_installed), 4.2 "Flow" (requirement before step-up, audit of denials), 7.4 "Query pattern" rule 1 (writes are outside the visibility lint; the estimate list of cost.reconcile.read has no visible-channel predicate, R-AZ-28).
docs/spec/03-data-model.md (core contract, read only): 4.9 "Costs (billing)" with price_table and cost_entries (every column, the partial unique index on supersedes_id, the indexes, "Valid total = entries without a successor", personal amounts are information only); runs (cost_micros, currency, cost_is_estimate, version, billing_mode_expected); agent_sessions (cumulative_cost_micros, cost_baseline_micros, currency, cost_is_estimate, version); 6.3 "Lock order"; R-DM-23.
docs/spec/02-architecture.md: 4.2 (module billing owns price_table and cost_entries), 4.3 (ports: billing Costs with "posting, correction, totals"), 4.4 (billing may use nothing; runs may use billing and agents), 4.5 (kernel/txn, kernel/auditing, kernel/authorize, kernel/measure), 4.6 (row billing with cost.reconcile.read), 6 "Code layout" (module layout, test/**, the list of places with the working name), 8.1 "Flow", 8.2 "Lock order", 8.3 "In-Tx events", 12.1 "Usage" (job arguments only IDs), 12.2 (queue default), 12.3 row billing.reconcile ("never periodic"), 12.4 "Rules for jobs", 13 "Configuration loading" point 3 (*_ref keys).
docs/spec/06-api.md: 4 point 2 (the exception for listCostEstimates, R-AZ-28), 5 "Errors" (closed code list, errors[], reason), 6 "Pagination" (cursor, limit, {items, next_cursor}), 7 "Idempotency" (POST with Idempotency-Key: replay, 409 request_in_progress, 422 idempotency_key_reuse; stored and not stored answers), 16 group "Stats" (rows reconcileCosts, submitCostReconciliation, listCostEstimates, getThreadCosts, getCostStats).
docs/spec/11-extensions.md: 1 "Principles", 2 "Agent or app", 3 "App manifest", 4 "Installation, update, uninstallation", 5 "Scopes and grants" (row costs:reconcile), 6 "Isolation and limits" (apps never run step-up actions), 10 "Ready-made integrations as examples" (own VERSION and CHANGELOG.md, fakes with fixtures from the provider documentation, no release component).
docs/spec/08-identity.md: 6.2 "Step-up flow", 12 "API tokens for agents, apps and tools".
docs/spec/16-operations.md 5.2 "Secret provider (ADR 0011)" (schemes file and env). docs/spec/19-security.md: SEC-SEC-1 to SEC-SEC-4, SEC-AUD-5 (audit_details.v1), SEC-NET-4 (no redirects), SEC-LOG-1 to SEC-LOG-7, SEC-T10. docs/spec/23-quality.md: 3 "Component contract" (Python project under plugins/), "Documentation in code" (ruff D, convention google), 11 (quality.sonar.new_coverage_min_percent).
docs/api/openapi.yaml: operations reconcileCosts, submitCostReconciliation, listCostEstimates, getThreadCosts, getCostStats, installApp, validateManifest, setAppScopes, uninstallApp, createToken; schemas CostReconcile, ReconcileEntry, CostReconcileResult, CostEstimate, CostEstimatePage, CostBreakdown, Money, AppScope, AppManifest, AppInstall, Problem (x-problem-codes), FieldError; parameters IdempotencyKey, Cursor, Limit; responses BadRequest, Unauthorized, Forbidden, Conflict, UnprocessableEntity, TooManyRequests, Error. No AsyncAPI channel and no webhook event carries costs (docs/api/asyncapi.yaml, docs/api/webhooks.md).
docs/operations/configuration.md: section 2 (duration format, secret_ref), section 4 rows reconciler.*, ratelimit.rl.admin.per_minute, ratelimit.rl.admin.burst, auth.step_up_max_age, auth.dpop_required_for, jobs.queue.default.max_workers, database.tx_retry_max, http.max_json_body_bytes, sync.idempotency_key_ttl, audit.deny_aggregate_window, sdk.http_timeout.
docs/observability/signals.md: 4.3 bothy.run.cost (attribute bothy.cost.estimate; not recorded by this ticket, note 3i), the job metrics bothy.job.duration and bothy.job.queue, the span job <bothy.job.kind>, the span txn.subscribe <Subscriber>.
docs/security/threat-model.md: T-PL-01, T-TK-02. docs/RISKS.md: RK-16, RK-17. docs/DECISIONS.md: D-BR-02, D-K-D28, D-W-21, D-W-100, D-W-103, D-W-104 (cost.reconcile.read, R-AZ-28), section 10 (format of the rows). docs/OPEN-QUESTIONS.md: no question applies to this ticket (checked).
docs/PLAN.md: section 3 "Way of working per ticket"; section 8 "M2 Agent platform, extensions and terminal" (Goal with P-18, Scope "costs and reconciliation" and ADR 0055, Exit criteria 2 and 10, Test strategy, Red team "exceeding the scopes of an app", Risks "the cost estimate differs from the invoice"); rows M2-22 (booking, price table, getThreadCosts), M2-24 (apps, installation, scopes, app tokens), M2-12 (agent sessions), M2-27 and M2-29 (the neighbours; the Python SDK is not in this ticket's closure), M2-28 (this ticket); the introduction of block "reference integrations" (no test calls a real service; fakes with fixtures from the provider documentation); M4-28 and M6-05 (console and statistics).
Dependencies: the header names M2-22 and M2-24. Implied through them: M2-12 (through M2-22, D-W-103; the job updates agent_sessions, 10 12.4 point 4), M2-09, M2-08, M2-07, M2-06, M2-05, M2-04, M2-02 and the M1 tickets M1-01 to M1-29, M1-31 to M1-37, M1-44 to M1-46, M1-48 to M1-56, M1-60 to M1-66, M1-68, M1-69 and M1-71. M1-70 is not in the closure and is only touched when it is already merged; M1-69 and M1-71 are in the closure and therefore merged before this brief starts (Design note 15).
M1 briefs whose interfaces this ticket uses: M1-05 (module layout, architecture/modules.yaml), M1-06 (make register-check), M1-07 and M1-08 (configuration schema in internal/wire, make config-doc-check), M1-12 (internal/platform/pg/pgtest), M1-13 (txn.UnitOfWork, txn.Publish, txn.Subscribe, the golden file of registrations, CheckTxnDirections), M1-14 (internal/platform/jobs: Args, Handler, Register, HandlerOptions, Inserter.InsertTx, ValidateArgsType), M1-16 (job adapter pattern internal/<module>/adapters/jobs), M1-17 (api/operations.yaml, strict server, route table), M1-18 (StageValidate with the FieldError code vocabulary, problem writer with reason), M1-19 (rl.admin), M1-20 (Idempotency-Key, Idempotent-Replay), M1-24 (auditing.Recorder.Record, auditing.Entry, Attrs with Time, Int, Code), M1-25 (registration duty), M1-28 (authorize.Authorizer.Authorize, authorize.Resource, requirement Owner for owner, AppKind for "app with scope costs:reconcile", Any(Owner, AppKind) for "owner or app with scope costs:reconcile" (R-AZ-28), the app scope table, decision to error mapping), M1-35 (step-up), M1-44 (API tokens), M1-46 (test of places with the working name, tools/ci/tests/test_product_name_places.py).
Code on the base branch (merged names win, D-W-100): services/server/internal/billing (M2-22), internal/apps (M2-24), internal/runs (M2-05, M2-09), internal/agents (M1-51, M2-07, M2-12), internal/authz, internal/kernel/{authorize,auditing,txn,ids,clock,errs,measure}, internal/platform/{jobs,pg,pg/pgtest}, internal/wire (configkeys_*.go, the txn golden file), api/operations.yaml, architecture/modules.yaml; plugins/README.md; .semgrep/bothy-go-no-model-credentials.yaml; the root Makefile (component discovery of Python projects under plugins/). Read merged M1 code with git show main:<path> if the base branch is older.

Fixed requirements that apply¶

Owner's fixed build rules (apply verbatim):

No MVP. New from scratch and directly in the final state; stages are only the order.
Strictly modular: clear module and service boundaries, dependencies only through defined interfaces, no cross access.
Documentation in code: every public module, class, function and interface with a docstring/doc comment (purpose, parameters, return value, errors, example where useful); comments explain the why, not the what. API documentation is generated from code (OpenAPI from the routes, dartdoc, Python API docs in MkDocs).
Documentation in the repo: README per module/service, ADR for every architecture decision, CHANGELOG per release, keep the documentation site current. A change without matching documentation is not done.
Quality gates (SonarQube, linters, type checking, security scanners, test coverage) block the merge.

Rules for every brief:

Test first: write the failing test, run it and keep the red output for the report, then make it green, then clean up.
No en dash and no em dash in any file you write; the text-rules pre-commit hook rejects them. Use a hyphen or rephrase.
White label: examples and test data use example.com and the people Alex, Sam and Robin; no company, host or personal names, no paths under a home directory.
Limits, windows and thresholds only through named configuration keys listed in docs/operations/configuration.md; code comments and docs name the key, never the number.
Specification first: a change to behaviour, API, event, right or configuration updates docs/spec/ or docs/api/ in the same change. The core contract (docs/spec/00-glossary.md, 03-data-model.md, 04-sync-protocol.md, 05-actions.md) is not edited by this ticket.

Product requirements implemented here:

The reconciliation is an app outside the server, off by default, with its own process, its own configuration and its own secret provider; the provider's admin key exists only there; the app needs no inbound port and talks to the server only through the public API (10 12.4, ADR 0055 points 1 and 4, 11 1, PLAN P-18).
The server never receives, reads, stores or forwards a provider key: no server configuration key, table column, job argument or code path for it, and no environment read of provider credentials (.semgrep/bothy-go-no-model-credentials.yaml; ADR 0055 "Confirmation"; B10, I5; PLAN row "the server stores no vendor key (canary)").
cost.reconcile (reconcileCosts): instance, owner only, step-up yes, audit yes, rl.admin, effect write; agent, app and personal tokens are refused (05 4.12, 2.4, R-AZ-03; 07 4.2).
cost.reconcile.submit (submitCostReconciliation): instance, only an app with the granted scope costs:reconcile, no step-up, audit yes, rl.admin; it only adds reconciled entries to existing runs and never changes or deletes another entry (05 2.6 and 4.12, ADR 0055 point 3 and "Confirmation", 10 12.4 "Authorization").
A reconciled entry has source = 'reconciled', is_estimate = false, reconciled_at and supersedes_id on the entry it replaces; every entry has at most one successor; the valid sum is the sum of the entries without a successor (03 cost_entries, 10 12.4 point 4).
Both ways apply their entries through the same job billing.reconcile in queue default, never periodic; job arguments carry only IDs; the job can run again without a second effect (02 12.1, 12.3, 12.4; PLAN row M2-28).
After the job, runs.cost_micros, runs.cost_is_estimate, agent_sessions.cumulative_cost_micros and agent_sessions.cost_is_estimate reflect the valid entries; amounts of different currencies are never converted (10 12.1 and 12.4 point 4).
The app distributes each day's amount of a workspace over the estimates of the mapped agents in proportion to the estimates; the sum equals the actual amount; the rounding remainder goes to the largest entry; amounts without a matching run are reported as not assigned (10 12.4 point 3, ADR 0055 point 2, BR-10).
The app asks the provider at most once per reconciler.min_request_interval, for the last reconciler.lookback days, and maps workspaces to agents through reconciler.workspace_map (10 12.4 points 1, 2 and 5; 10 17).
The audit entry of a reconciliation holds the period, the numbers of entries and the sums per currency, never a secret or free text (10 14; 19 SEC-AUD-5, SEC-SEC-3).
Errors follow 06 "Errors" with codes from x-problem-codes; both operations accept Idempotency-Key with the behaviour of 06 "Idempotency".
Amounts of runs in mode personal are information, never invoice (03 cost_entries, 10 12.2); they are never reconciled (Design note 4).
cost.reconcile.read (listCostEstimates): instance, an owner in a session or an app with the granted scope costs:reconcile, no step-up, audit no, effect read, rl.read; per run it returns only run, agent, provider, currency, the estimate and whether it is reconciled, of mode organization, never a channel, a thread or content, and it is not filtered by visible channels; agents, personal tokens and people who are not owners get 403 forbidden (05 rule 1.2, 2.6, 2.7, 4.12 and R-AZ-28; 06 4 point 2).

Scope¶

All server paths are under services/server/ unless noted. This ticket creates no make target and no migration.

Create (server):

internal/billing/domain/reconcile.go: pure rules on domain types (the domain may not import the port, 02 5), standard library and kernel-core only: types Request, RequestEntry, Entry (one row of cost_entries), Share, Violation (pointer and code), UnmatchedReason; functions Validate(req Request) []Violation (Design note 3b), Match(valid []Entry, req Request, e RequestEntry) (matched bool, noop bool, reason UnmatchedReason) (note 4), Split(amount int64, valid []Entry) []Share (note 5). The app layer converts the port DTOs.
internal/billing/app/reconcile.go: the use case Reconcile for both actions (note 3) and the outgoing port ReconcileScheduler (inserts the job).
internal/billing/app/apply.go: the job use case ApplyReconciliation(ctx, runIDs []ids.ID) error (note 6).
internal/billing/adapters/postgres/queries/reconcile.sql: take the per-run advisory locks, read the valid entries of runs in a period, insert reconciled rows, totals per run, totals per session (notes 3 and 6); plus the query of the read operation of note 12.
internal/billing/adapters/jobs/: args Reconcile{RunIDs []ids.ID} with kind billing.reconcile, its handler (queue default, no periodic registration), the ReconcileScheduler implementation with jobs.Inserter.InsertTx.
internal/runs: subscriber OnCostsReconciled in internal/runs/app (or the existing subscriber file) and the query that sets the cost totals of runs (note 7).
test/reconcile/: the end-to-end test of note 14 (reconcile_e2e_test.go), its fake cost API, and README.md.

Create (app, all under plugins/cost-reconciler/):

pyproject.toml, uv.lock, README.md (English, like the other READMEs under plugins/), VERSION, CHANGELOG.md, manifest.json.
Package cost_reconciler/: __init__.py, __main__.py (command line), config.py, secretref.py, amounts.py, costapi.py, server.py, distribute.py, reconcile.py (one pass), logs.py, uuid7.py (note 11).
tests/: fakes.py (fake cost API and fake server on 127.0.0.1 port 0), fixtures/ (cost report pages), and the tests of the section "Tests".

Change:

internal/billing port: Costs.Reconcile(ctx context.Context, in ReconcileInput) (ReconcileResult, error) with ReconcileInput{Action authorize.Action; Provider string; PeriodStart, PeriodEnd time.Time; Entries []ReconcileEntry}, ReconcileEntry{RunID ids.ID; AmountMicros int64; Currency string}, ReconcileResult{Reconciled int; UnmatchedRunIDs []ids.ID}; In-Tx event CostsReconciled{Runs []RunCostTotal; Sessions []SessionCostTotal} with RunCostTotal{RunID ids.ID; AmountMicros int64; Currency string; IsEstimate bool} and SessionCostTotal{SessionID ids.ID; AmountMicros int64; Currency string; IsEstimate bool}; the read method of note 12. Merged names of M2-22 win where they differ only in name.
internal/billing/adapters/httpapi: the strict-server methods for reconcileCosts, submitCostReconciliation and the read operation of note 12.
internal/agents port Sessions: ApplyCostTotals(ctx context.Context, totals []SessionCostTotal) error with the agents' own type (no import of billing), unless M2-22 already added an equivalent that sets totals; then use that one.
internal/wire: register the job handler for the roles worker and all, the scheduler for all roles, txn.Subscribe for runs.OnCostsReconciled; regenerate the golden file of In-Tx registrations; the test of note 3k.
internal/authz: tests of the requirements of cost.reconcile, cost.reconcile.read and cost.reconcile.submit; the entry of the read action cost.reconcile.read (requirement Any(Owner, AppKind) for the 05 text "owner or app with scope costs:reconcile", scope table row costs:reconcile; R-AZ-28), unless the base branch has it already.
architecture/modules.yaml: cost.reconcile.read under billing as 02 4.6 lists it, unless it is there already (make register-check).
api/operations.yaml: move reconcileCosts, submitCostReconciliation and listCostEstimates (note 12) from later to enabled; rerun the route generator.
docs/api/openapi.yaml: the changes of note 10.
docs/spec/10-backend-policy.md: 12.4 (how the server stores, matches and applies; note 16) and 17 (the new app keys of note 11c); docs/spec/02-architecture.md: 4.3 rows billing and runs (the In-Tx event), 12.3 row billing.reconcile (note 6); docs/operations/configuration.md regenerated with make config-reference.
docs/adr/0055-cost-reconciliation-as-its-own-app-admin-key-never-in-the-server.md: add the names of the tests of this ticket to "Confirmation" without changing the decision.
READMEs: internal/billing/README.md (port, event, job, the request and job split; replace "periodic, if enabled" with "never periodic" as 02 12.3 says), internal/runs/README.md, internal/agents/README.md, test/reconcile/README.md, plugins/README.md (row for cost-reconciler/), plugins/cost-reconciler/README.md.
services/server/CHANGELOG.md (Unreleased), plugins/cost-reconciler/CHANGELOG.md, docs/DECISIONS.md section 10 (rows D-M2-28-<n>).
As Design note 15 says: the root Makefile (one line of typecheck, only under its condition), test/canary (M1-69, in the closure), the visibility lint exception list (M1-70, only if it is merged).

Out of scope¶

Booking of costs at the end of a run, the price table, getThreadCosts, listPrices, createPrice, closePrice, the label "Estimate" and BR-09, BR-11 (M2-22).
App manifests, installApp, validateManifest, scopes in authz, app tokens from createToken, setAppScopes, uninstallApp (M2-24); this ticket only uses them.
The table agent_sessions and the port agents.Sessions itself (M2-12).
Cost statistics getCostStats (M6-05); the console views for costs and a manual reconciliation form (M4-28 and the console tickets of M6).
The Python SDK (M2-29, not in this ticket's closure); the app uses the public REST API directly.
BR-13 for the adapters (M2-33) and the runner's key handling (M2-34).
The canary test across all outputs SEC-T10 (M1-69), the SQL visibility lint (M1-70) and the Schemathesis contract tests (M1-71); this ticket only extends them as note 15 says (M1-69 and M1-71 are in the closure, M1-70 only if it is merged).
The maintenance mode layer of 05 rule 1.9 (M7, P-17); the reconciliation actions have effect write and fall under it when it comes.
A release component and packaging of the app (18, M7 release chain); like the reference apps of 11 10, operators build it from the repository.
Reconciliation apps for other providers (ADR 0055: each is another app) and costs of a priority tier, which the provider's cost endpoint does not report (ADR 0055, "Consequences").
Changes to the core contract.

Boundaries¶

Work in a dedicated git worktree on the branch feat/M2-28-cost-reconciliation; one writer per worktree.
Conventional Commits with the module as scope (for example feat(billing): ..., feat(runs): ..., feat(cost-reconciler): ..., docs(api): ...) and a DCO sign-off on every commit (git commit -s); squash into meaningful commits before the merge (docs/quality/push-discipline.md).
Never push to main, never tag and never create remote repositories; the only remote steps are the branch push and the pull request of AGENTS.md.
Do not modify files outside the scope above, except the CHANGELOG, README and docs/DECISIONS.md entries named in this brief.
Follow the host rules supplied by the launcher (ports, machine rules).
Tests bind only 127.0.0.1 and port 0; PostgreSQL runs only as a throwaway container on a random port through pgtest (P-09, P-10). Without a Docker-compatible runtime the integration tests fail; they never skip.
Stop only processes you started, by PID. Temporary directories only via mktemp -d, removed at the end.
Never read or print secrets. No network calls except package downloads for pinned tools, Go modules and Python packages pinned in uv.lock.
Never lower a threshold, disable a gate or delete a test.
If the core contract lacks something this ticket needs, stop with Verdict: BLOCKED <reason> instead of building a deviation (P-11).
This brief was written before milestone M1 closed (D-W-100). Where the merged code of an M1 ticket names a package, type, function, file or make target differently from this brief, follow the code and list each difference in the report; where it behaves differently from the specification, stop with Verdict: BLOCKED <reason>.
No test, script or manual step calls the provider's real API or uses a real admin key; the provider side is a fake on 127.0.0.1 with fixtures written after the provider documentation cited in 10 12.4. A live run against a real organisation needs an admin key and an external account and is the owner's decision, outside this ticket.
Admin keys and app tokens in tests are random canary values created at run time; no file in the repository contains a key-like value (make gitleaks). No model runs and no agent runtime starts in any test.
The server gets no configuration key, column, job argument, log field or code path that could hold a provider key; never add an exception to .semgrep/bothy-go-no-model-credentials.yaml.
The app's Python dependencies are pinned in its uv.lock and pass make licenses; prefer the standard library. Its virtual environment and caches stay inside the worktree or a mktemp -d folder.
Changes to code of merged tickets (M2-22, M2-24, M2-12, M2-09, M1-28) stay within the Design notes, each with a test; no other behaviour of those tickets changes.

Design notes¶

Preconditions. Check them before writing any code; if one fails, stop with the verdict named. a. Read action for the estimates (core contract, R-AZ-28 under D-W-104): 05 4.12 row cost.reconcile.read (resource instance, right "owner or app with scope costs:reconcile", Owner, Step-up, Audit and Permissions no, class rl.read, effect read by 2.7), 05 2.6 (scope costs:reconcile), rule 1.2 (the list is evaluated on the instance without a visible-channel predicate), 02 4.6 (module billing) and in docs/api/openapi.yaml the operation listCostEstimates with the schemas CostEstimate and CostEstimatePage; build it as note 12 with exactly these values. A base branch whose docs/spec/actions.yaml lacks cost.reconcile.read (with app_scope: costs:reconcile) or whose docs/api/openapi.yaml lacks listCostEstimates has not taken over that spec state: stop with Verdict: BLOCKED spec state: cost.reconcile.read (05 4.12, R-AZ-28) or listCostEstimates missing on the base branch. b. agent_sessions and the port agents.Sessions of M2-12 (implied through M2-22, D-W-103) exist on the base branch; if not, stop with Verdict: BLOCKED dependency M2-12 (agent_sessions) not merged. c. The merged cost_entries has what 03 defines and this ticket uses: is_estimate, source with reconciled in its check, reconciled_at, supersedes_id with its foreign key and the partial unique index (supersedes_id) WHERE supersedes_id IS NOT NULL, the indexes (run_id), (thread_id) and (agent_id, occurred_at). A missing part is a deviation of M2-22 from 03: stop with Verdict: BLOCKED per the D-W-100 boundary.
Shape across modules. billing may use no other module (02 4.4), but the totals live in tables of runs and agents. So: the request writes only cost_entries (owned by billing) and inserts the job; the job reads cost_entries, computes totals and publishes the In-Tx event billing.CostsReconciled; runs (which may use billing and agents) subscribes, updates its own rows and hands the session totals to agents.Sessions.ApplyCostTotals. Every edge points downward; CheckTxnDirections of M1-13 accepts it. Ruling for docs/DECISIONS.md, with 02 4.3 updated (row billing "Emits (in-tx)" CostsReconciled; row runs "Subscribes (in-tx)" billing.CostsReconciled). Reason: the same pattern as agents.AgentDisabled; no new module edge.
Request use case Costs.Reconcile (both operations call it with their action), inside the request's unit of work: a. authorize(actor, action, Resource{Kind: instance}) first. If the generated route stage of M1-28 already authorizes instance-level actions, do not call it twice (follow the merged code). Denials are audited by the authz layer (audit yes, R-AZ-04). b. Rules after the schema check of StageValidate, all violations at once as 400 validation_failed with errors[] (codes from the M1-18 vocabulary): period_start not before period_end gives invalid at /period_end; a run_id that occurs again gives invalid at /entries/<i>/run_id of the later occurrence. The schema checks of note 10 give too_small for a negative amount_micros, pattern_mismatch for a currency that is not three upper-case letters and too_short for an empty provider. c. Lock: for each distinct run ID of the request, sorted ascending in Go and one call per ID, SELECT pg_advisory_xact_lock(hashtextextended('billing.reconcile:' || $1::text, 0)). This is the domain step of 03 6.3 rule 2, the same pattern as M1-32 and M3-03. No row lock: bothy_app has only SELECT, INSERT on cost_entries (M2-22 note 2c), and FOR UPDATE/FOR SHARE need the UPDATE privilege. The partial unique index on supersedes_id stays the backstop. d. In a new statement after the lock (READ COMMITTED sees what a concurrent reconciliation committed meanwhile), read the valid entries (no other entry has supersedes_id on them) of these runs with occurred_at >= period_start AND occurred_at < period_end. e. Match each entry in request order (note 4). Unmatched run IDs go to unmatched_run_ids in request order. f. For each matched run that is not a no-op: compute the shares (note 5) and insert one row per valid entry in the period: new UUIDv7 id, the copies run_id, agent_id, thread_id, channel_id, space_id, agent_session_id, billing_mode, provider, model and all token columns of the replaced entry, occurred_at of the replaced entry (the day stays the same), amount_micros the share, currency of the request entry, is_estimate = false, source = 'reconciled', reconciled_at from kernel/clock, supersedes_id the replaced entry. Only inserts; no UPDATE and no DELETE on cost_entries anywhere in this ticket. g. If at least one run got new rows: InsertTx one job billing.reconcile with these run IDs in ascending order. Both actions insert the same kind with the same args type. h. Audit allow entry in the same transaction through auditing.Recorder.Record: action cost.reconcile or cost.reconcile.submit, resource kind instance without ID, Attrs: period_start and period_end (Time), entry_count, reconciled_count and unmatched_count (Int), and one member total_micros_<currency in lower case> (Int) per currency with the sum of the amounts of the matched entries. Nothing else; no provider name, no run IDs (10 14: "period, number of entries, sums per currency"). Ruling for docs/DECISIONS.md: the same details for cost.reconcile, which 10 14 does not list separately. i. This ticket records no bothy.run.cost point for reconciled entries: billing may use no other module (02 4.4) and so cannot set the required attribute bothy.agent.handle (signals.md 4.3; cost_entries holds only agent_id, 03), the metrics port that M2-22 adds for bothy.run.cost belongs to runs (M2-22 note 5 h), and metrics are recorded only after the unit of work committed, never inside it (M2-05 note 11). Name the missing bothy.cost.estimate = false points as a spec gap in the report. j. Answer 200 with CostReconcileResult{reconciled, unmatched_run_ids}: reconciled counts the matched entries (written or no-op), so reconciled plus the number of unmatched IDs equals the number of entries; unmatched_run_ids is [] when empty (ADR 0053). k. A test in internal/wire asserts that the server's configuration schema has no key under reconciler. and no key whose name contains admin_key (the reconciler.* keys have the place "Reconciliation app", 10 17).
Matching (Ruling for docs/DECISIONS.md, documented in 10 12.4). A request entry for run R is matched when R has at least one valid entry in the period and all of them have billing_mode = 'organization', provider equal to the request's provider and currency equal to the entry's currency. Otherwise R is unmatched: no cost entry at all (also an unknown or still active run; billing cannot and need not tell them apart), no valid entry in the period, mode personal (amounts are information, never invoice, 03), another provider, another currency or mixed currencies (no conversion, 10 12.1). A matched run is a no-op when all its valid entries in the period are reconciled and their sum equals the amount: nothing is written and it counts as reconciled. Reason: the app submits the same days again on every pass within reconciler.lookback; the second pass must not grow the chain. A different amount supersedes the reconciled entries (a correction, "a correction entry replaces an earlier one", 03). Entries of R outside the period stay as they are (a run with turns on two days is reconciled per day).
Split within a run (Ruling for docs/DECISIONS.md). One request entry carries one amount per run, but a run has one entry per model and turn (10 12.1, 09 4.10). The amount A is split over the run's valid entries in the period in proportion to their current amounts: share = floor(A times amount_j divided by the sum); the remainder goes to the entry with the largest amount, ties to the smallest id; if all amounts are 0, the whole of A goes to the entry with the smallest id. Every valid entry in the period gets exactly one successor, also with share 0, so no estimate stays valid next to a reconciled amount. Compute without int64 overflow (math/big or math/bits). Reason: the same rule as BR-10 one level down; the per-model attribution is a proportion, as the ADR states for the day.
Job billing.reconcile (02 12.3, queue default, River defaults, never periodic): the handler in internal/billing/adapters/jobs calls ApplyReconciliation(ctx, runIDs) in one unit of work: a. For each run ID of the job in ascending order SELECT pg_advisory_xact_lock_shared(hashtextextended('billing.reconcile:' || $1::text, 0)). With the exclusive lock of note 3c this orders a job against a concurrent request: a request that commits new rows after the job read the totals had to wait for the job, and its own job runs afterwards. b. Per run: amount = sum of the valid entries (all periods), currency = their currency (one currency per run after note 4; if a run's valid entries mix currencies, leave that run unchanged, log the run ID without amounts and go on), is_estimate = any valid entry has is_estimate = true. c. Per agent_session_id of these runs: the same over all valid entries of the session (query by thread_id and agent_session_id, index (thread_id)). d. txn.Publish(ctx, billing.CostsReconciled{Runs, Sessions}), both lists in ascending ID order. e. Ruling for docs/DECISIONS.md, with 02 12.3 updated ("the request stores the reconciled entries; the job applies them to runs and sessions"): the request stores the entries because job arguments carry only IDs (02 12.1; ValidateArgsType of M1-14 allows only ids.ID, []ids.ID and int64) and 03 has no other table that could hold a submission; the job applies them. The job reads its state again on every run, so a repeated run has no second effect (02 12.4 rule 1).
Totals in runs and agents. runs.OnCostsReconciled updates, in ascending id, cost_micros, currency and cost_is_estimate of each run whose values differ and raises version by one for each changed row (03: version "rises with every stored change"); unchanged rows are not touched. Then it converts the session totals to the agents' type and calls agents.Sessions.ApplyCostTotals, which sets cumulative_cost_micros and cost_is_estimate where they differ, in ascending id, raises version by one for each changed row (03 principle 10, as M2-22 Design note 5 g does when it books; an adapter whose next agent_session.report then gets 412 version_conflict receives the row in current, 06 5) and leaves cost_baseline_micros alone. No durable event is written: 05 section 6 has no event for costs; clients see the reconciled amounts on the next read (getThreadCosts, getRun; spec gap reported).
Authorization (07 4.2 order: requirement before step-up):
- reconcileCosts: an owner in a session with a step-up younger than auth.step_up_max_age is allowed; an owner without a fresh step-up gets 401 step_up_required; an app token (also with costs:reconcile), an agent token and an owner's personal token fail the requirement Owner (owner rights need a session, 05 2.3 point 1) and get 403 forbidden with reason: owner_only; a non-owner person gets the same. This is how R-AZ-03 holds here: no token reaches the step-up step. Nothing is written, no job is inserted, the denial is audited.
- submitCostReconciliation: an app with costs:reconcile in granted_scopes is allowed; an app without it gets 403 scope_missing; a person (also an owner) and an agent get 403 forbidden with reason: subject_kind_not_allowed (requirement AppKind, M1-28); an uninstalled app gets 401 (app_not_installed).
- listCostEstimates (cost.reconcile.read, R-AZ-28): an owner in a session (no step-up) and an app with costs:reconcile in granted_scopes are allowed; an app without it gets 403 scope_missing; an agent token, an owner's personal token and a non-owner person get 403 forbidden with reason: owner_only (requirement Any(Owner, AppKind), whose failure names its first alternative, M1-28); an uninstalled app gets 401 (app_not_installed). The allowed read writes no audit entry (Audit no); every denial is audited (R-AZ-04).
- Agents never hold costs:reconcile (05 2.6); no agent path reaches any of the three actions.
Idempotency and concurrency. Idempotency-Key is handled by the stage of M1-20: a replay returns the stored answer with Idempotent-Replay: true and inserts no rows and no job; another body under the same key gives 422 idempotency_key_reuse; a repeat while the first runs gives 409 request_in_progress; a 401 step_up_required is not stored, so the same key works after the step-up. Two concurrent requests for the same run serialize on the locks of note 3c; afterwards every replaced entry has exactly one successor and the valid sum equals the amount of the request that committed last. Deadlocks are retried by the unit of work (database.tx_retry_max).
API contract fixes in docs/api/openapi.yaml (spec first; regenerate with the generator of M1-17): reconcileCosts gets the response '409' (Conflict, for request_in_progress of 06 7); submitCostReconciliation gets '422' (UnprocessableEntity, for idempotency_key_reuse); CostReconcile.provider gets minLength: 1; ReconcileEntry.amount_micros gets minimum: 0; ReconcileEntry.currency gets pattern: '^[A-Z]{3}$' with the description "ISO 4217 code" as Money.currency; the descriptions of both operations state the matching and the period rule of note 4 and that the entries are applied by the job. The read operation listCostEstimates of note 12 is already in the contract (note 1a). Keep summary texts unchanged so the generated tables of 06 do not change; if they do, run make api-tables and commit the result. make api-lint and make api-checks stay green.
The app plugins/cost-reconciler (Ruling for docs/DECISIONS.md: name, language and shape). A Python project, because the root Makefile discovers Python projects under plugins/ and runs format, lint, bandit, mypy, pytest with coverage and the license gate on them (23 3); flat layout with the package cost_reconciler and [tool.uv] package = false like tools/pyproject.toml, started as python -m cost_reconciler; standard library for HTTP (urllib.request, ssl), TOML (tomllib) and decimals. The working name of the product appears in none of its files (02 6, M1-46 test of places with the working name). It uses the public API only, without the SDK (M2-29 is not in the closure). a. pyproject.toml: requires-python as tools/pyproject.toml; dev group with ruff, mypy, pytest, pytest-cov pinned like in tools/; ruff rules including D with convention google (tests excluded); mypy strict over the package and the tests; pytest with filterwarnings = ["error"]; coverage with branches and fail_under at the value of quality.sonar.new_coverage_min_percent (23 11 names no key for Python components; the whole app is new code; spec gap reported). b. Command line: python -m cost_reconciler --config <file> [--once]. Without --once it repeats passes until SIGTERM or SIGINT, which end it after the current request. Exit codes: 0 success, 1 a pass failed (only with --once), 2 configuration error. Every request to the cost API, also across passes, starts at least reconciler.min_request_interval after the previous one (monotonic clock, injectable for tests). The README says that a schedule running --once must not start more often than that interval. c. Configuration: a TOML file with the table [reconciler] and the sub-table [reconciler.workspace_map] (workspace ID to a list of agent IDs). Keys: the three of 10 17 and, Ruling for docs/DECISIONS.md with rows added to 10 17 (place "Reconciliation app") and make config-reference: reconciler.server_url (base address of the server API, no default, required), reconciler.token_ref (secret_ref of the app token, no default, required), reconciler.admin_key_ref (secret_ref of the provider admin key, no default, required), reconciler.cost_api_url (base address of the provider API from the documentation cited in 10 12.4, no default, required, so no host name is built into the code) and reconciler.http_timeout (time limit of one HTTP request, default the standard value of sdk.http_timeout). Unknown keys, a missing required key, a duration outside the format of configuration.md 2, a reconciler.lookback that is not a whole number of days, an empty reconciler.workspace_map and an agent ID listed under two workspaces are configuration errors that name the key, never a value. Both addresses must use https, except http to a loopback address (tests, a local proxy). d. Secrets (secretref.py, 16 5.2): file:<absolute path> reads a file with mode 0400 or 0600 owned by the process user, else a configuration error (SEC-SEC-1); env:<NAME> reads a variable (allowed, not recommended); other schemes are a configuration error. The value is wrapped in a type whose repr and str are a fixed redaction mark; only the HTTP clients unwrap it to build a header. Secrets are read again at the start of each pass, so a rotation needs no restart. A logging filter replaces the current secret values in every record with the redaction mark (defence in depth). e. Cost API client (costapi.py, 10 12.4 point 2): GET <reconciler.cost_api_url>/v1/organizations/cost_report for the window of complete UTC days from the start of today minus reconciler.lookback up to the start of today, daily buckets, grouped by workspace_id and description, authenticated with the admin key in the way the provider documentation cited in 10 12.4 (Q2) prescribes; follow the paging of the response to the end. Timeout reconciler.http_timeout; no redirects (a 3xx is an error, so the key never goes to another host, SEC-NET-4 applied to the app); a status other than 2xx is an error that carries the status only, never the response body. A page without the expected bucket or line fields fails the whole pass before any submission (never submit from a partial report). Unknown fields are ignored. Write the fixtures after the provider documentation; the shape facts the spec states are daily buckets, grouping by workspace and description, costs in USD as a decimal string in cents. List the fields you relied on in the report under "What is not proven". f. Amounts (amounts.py): per bucket and workspace and currency, sum the decimal strings exactly (decimal.Decimal), then convert cents to micro units once (ten thousand micro units per cent) with ROUND_HALF_EVEN to an integer. An invalid decimal string fails the pass. g. Estimates (server.py): for each bucket, one call of the read operation of note 12 with from and to equal to the bucket bounds and the agent IDs of all mapped workspaces, following its cursor; group the items by the workspace of their agent. h. Distribution (distribute.py, pure, BR-10; Ruling for docs/DECISIONS.md): input the bucket amount T of one workspace and currency, and the estimates of the runs of its agents. Runs with another currency are left out and counted as currency_mismatch; runs with estimate 0 are left out (they stay estimates). No run left: the whole T is not assigned (no_runs); also when the remaining estimates sum to 0 (zero_estimates, the proportion is undefined). Otherwise share_i = floor(T times e_i divided by the sum of e), and the remainder T minus the sum of shares goes to the run with the largest estimate, ties to the smallest run ID. Workspaces without an entry in the map are not assigned (no_mapping). Integer arithmetic only. i. Submission (server.py): one POST <reconciler.server_url>/api/v1/costs/reconciliations per bucket with at least one entry: Authorization: Bearer <token>, Idempotency-Key a new UUIDv7 (uuid7.py, RFC 9562, because the standard library of the supported versions may lack it), body CostReconcile with provider the provider name the runtimes report for this provider in cost_entries.provider (the provider example of CostReconcile in openapi.yaml), period_start and period_end the bucket bounds, the entries. No redirects. 200: log reconciled and the unmatched run IDs. 400: log the pointers and codes of errors[], fail the pass. 401, 403: log status and problem code and reason, fail the pass. 409, 422, 429, 5xx and network errors: log and fail the pass; the next pass submits again and the server's no-op rule (note 4) prevents a second effect. No retries inside a pass (no SDK retry keys apply to the app). j. Logs (logs.py): JSON lines on stderr with ts, level, event and fixed fields (bucket start, workspace ID, currency, amounts in micro units, counts, run IDs, status, problem code); events pass_started, pass_finished, pass_failed, not_assigned (with reason no_mapping, no_runs, zero_estimates), currency_mismatch, submitted, unmatched_runs, config_error. Never a header, a token, a key, a request body or a response body of the cost API. This is how the app reports an amount without a matching run as "unassigned" (10 12.4 point 3). k. Manifest (manifest.json) in the format M2-24 validates (app_manifest.v1, schema AppManifest; if the merged validator follows 11 3.1, scopes.required): schema_version 1, slug cost-reconciler, name, description, publisher a neutral value as in 11 3.3, version equal to VERSION, exactly the scope costs:reconcile, no events, no delivery, no commands, no components, no requested grants. The README describes the setup: the owner installs the manifest with installApp (step-up, reason) granting costs:reconcile, creates the app token with createToken (shown once), stores token and admin key as files with mode 0600 for the app's service user, and runs the app; DPoP is not used (Ruling for docs/DECISIONS.md: bearer only, because auth.dpop_required_for does not name apps by default and the token reaches only the two cost actions cost.reconcile.read and cost.reconcile.submit; with app in that list the app cannot work, which the README says).
Read operation for the estimates (ids, right and class from the catalogue). GET /api/v1/costs/estimates (listCostEstimates) with the query parameters from and to (required timestamps, to exclusive, as getCostStats), agent_id (required, repeatable) and cursor and limit per 06 "Pagination"; x-authz-action the read action; x-rate-limit-class its class. Items per run with valid or replaced entries of mode organization and occurred_at in the range: run_id, agent_id, provider, currency, estimate_micros (sum of the run's estimate entries in the range whose successor, if any, is not an estimate: the estimate before any reconciliation) and reconciled (all valid entries in the range are reconciled). Sorted by run_id. Runs with mixed currencies in the range are left out. The query uses the index (agent_id, occurred_at). Instance-level, so no visible-channel predicate and no per-item check such as run.read (05 rule 1.2 and R-AZ-28, 07 7.4 rules 1 and 3); if the lint of M1-70 exists, add the query to its exception list with this reason. If the catalogue entry or listCostEstimates differs from this note, follow the contract and name the difference in the report.
App, token and installation in tests. Tests install the app through installApp with the file plugins/cost-reconciler/manifest.json (owner Alex with step-up), create its token through createToken (M2-24) and use the token only through the app or the fake app client; scope removal goes through setAppScopes, removal of the app through uninstallApp. Test worlds built through the API raise ratelimit.* in their configuration (R-QA-11).
End-to-end and canary (test/reconcile, Go, real server through internal/wire with role all so River works the job, pgtest, HTTP on 127.0.0.1 port 0). A fake cost API in Go serves the fixture pages of plugins/cost-reconciler/tests/fixtures/ and records every request. The test writes a TOML configuration and two secret files (mode 0600) into t.TempDir(): the app token, and as admin key a random alphanumeric canary created for this run. It starts the real app as a subprocess, uv run --directory <repo>/plugins/cost-reconciler --locked python -m cost_reconciler --config <file> --once, waits for exit code 0 and then until no billing.reconcile job is pending. A wrapper around the server's handler records every request and response (method, path, query, headers, body); the server's logs go to a buffer. Afterwards the canary must appear in none of: any row of any base table of the domain schema and of jobs.schema (one static statement over pg_class that applies query_to_xml(format('SELECT * FROM %I.%I', ...)) per table and searches the text with strpos; no SQL is built in Go, rule bothy-go-sql-string-building; the canary is alphanumeric so XML escaping cannot hide it), the recorded server traffic, the server logs, the app's stdout and stderr. Positive control: the fake cost API received the canary in the authentication header of every request, so the scan cannot pass for the wrong reason. uv missing is a failure, never a skip.
Conditional extensions. services/server/test/canary (M1-69) is in the closure: add reconcileCosts, submitCostReconciliation and the read operation to its scenario with a canary in provider (SEC-T10: "The ticket that builds a new path ... extends the scenario"). If the visibility lint of M1-70 is merged, add the billing read queries of this ticket to its exception list with the reasons of notes 3 and 12. If make typecheck cannot type-check the app from the repository root (it runs uv run --project <dir> --locked mypy in the root, and mypy reads its configuration from the working directory), change only that line to run inside the project directory as the test recipe does (cd $$p && $(UV) run --locked mypy) and name it in the report; if an earlier ticket already did, change nothing.
Documentation. 10 12.4 gets the server side in English: the request stores the reconciled entries after the matching rule and the split rule (notes 4 and 5), the period rule, the no-op rule, the job that applies them, and the app's reporting of unassigned amounts; 10 17 gets the new keys of note 11c. 02 4.3 and 12.3 as in notes 2 and 6. internal/billing/README.md: port with Reconcile and the read method, event CostsReconciled, job billing.reconcile (never periodic), lock order (advisory key namespace billing.reconcile:<run_id>), the queries without visible-channel predicate and their reason. internal/runs/README.md: subscription billing.CostsReconciled. internal/agents/README.md: ApplyCostTotals. plugins/README.md: one row (cost-reconciler/, app, cost reconciliation of 10 12.4, built by M2-28). test/reconcile/README.md: what the test proves and its prerequisites (uv, Docker).
Rulings for docs/DECISIONS.md (one row each, D-M2-28-<n>, with date, decision, reason and place): notes 2, 3h, 4, 5, 6e, 7 (no agent_session.changed for reconciled totals, as M2-22 for bookings), 11 (name, language and shape), 11a (coverage key), 11c (new app keys), 11h (distribution edge cases), 11k (bearer only), and the precondition handling of note 1.

Tests (write them first)¶

Unit (Go):

internal/billing/domain: TestValidateReconcile: period start equal to and after end, a repeated run ID (pointer on the later one), several violations reported together. TestMatch: a table with no entry, no entry in the period, mode personal, another provider, another currency, mixed currencies, an estimate in the period, entries partly outside the period (only the inside counts), reconciled with the same amount (no-op), reconciled with another amount (supersede). TestSplit: one entry, proportional shares summing to A, the remainder on the largest entry, a tie broken by the smallest ID, all amounts 0 (all on the smallest ID), A 0, values near the int64 limit without overflow.
internal/authz: TestCostReconcileRequirement: owner with a session and a fresh step-up allow; owner with an old step-up step_up_required; app token with costs:reconcile, agent token, owner's personal token and a non-owner person all forbidden owner_only. TestCostReconcileSubmitRequirement: app with the scope allow; app without it forbidden scope_missing; owner session and agent forbidden subject_kind_not_allowed. TestCostReconcileReadRequirement (R-AZ-28): owner with a session and no step-up allow; app with the scope allow; app without it forbidden scope_missing; agent token, owner's personal token and a non-owner person forbidden owner_only.
internal/wire: TestServerConfig_HasNoReconcilerKeys (note 3k); the txn golden file contains billing.CostsReconciled -> runs and CheckTxnDirections passes; TestJobs_BillingReconcileNotPeriodic: the jobs registry has the kind billing.reconcile in queue default and no periodic entry for it.

Integration (PostgreSQL 18 via pgtest, the server connects as bothy_app (pgtest Config(pg.RoleApp)), never as bothy_owner, real modules through internal/wire, HTTP through the real chain, people from the identity test helpers, the app installed as in note 13, runs and estimates created through the booking path of M2-22 and the run helpers of M2-09; agents A1 and A2 in mode organization, A3 in mode personal with holder Alex):

TestReconcileCosts_OwnerWithStepUp: Alex with a fresh step-up reconciles two runs: 200 with reconciled 2, new rows exactly as note 3f, one job billing.reconcile with both run IDs; before the job runs.cost_micros is unchanged, after it the totals and cost_is_estimate = false hold.
TestReconcileCosts_StepUpRequired: without a fresh step-up 401 step_up_required, nothing written; the same request with the same Idempotency-Key after the step-up gives 200.
TestReconcileCosts_TokensRejected (R-AZ-03): the app token with costs:reconcile, an agent token and Alex's personal token each get 403 forbidden with reason: owner_only; no row, no job; one audit deny entry each (aggregation per audit.deny_aggregate_window respected).
TestSubmitCostReconciliation_App: the app with the scope gets 200; the audit allow entry has exactly the details of note 3h.
TestSubmitCostReconciliation_ScopeMissing: after setAppScopes without costs:reconcile: 403 scope_missing, nothing written.
TestSubmitCostReconciliation_PeopleAndAgentsRejected: Alex's session and an agent token get 403 forbidden with reason: subject_kind_not_allowed; an uninstalled app gets 401.
TestReconcile_BothPathsUseSameJob (plan row): one request per path each inserts exactly one River job of kind billing.reconcile in its own transaction (a rolled back request inserts none); the totals change only after the job ran; both jobs run the same handler.
TestReconcile_BR10_EstimatesSuperseded (BR-10, server part): for a run with two models and a turn on another day, every valid entry in the period has exactly one successor with source = 'reconciled', is_estimate = false, reconciled_at set and copied tokens; the valid sum in the period equals the submitted amount; the remainder sits on the largest entry; the entry outside the period is untouched; getThreadCosts shows the run with the new amount and is_estimate = false after the job.
TestReconcile_OnlyAddsReconciledEntries (ADR 0055 "Confirmation"): a byte-equal snapshot of every row of cost_entries before the request is still present afterwards; all new rows are reconciled and point to an entry of a submitted run; bothy_app still has no UPDATE or DELETE privilege on cost_entries (has_table_privilege, has_any_column_privilege).
TestReconcile_Unmatched: an unknown run ID, a run without entries in the period, A3's run (personal), a run of another provider and an entry in another currency are listed in unmatched_run_ids in request order; reconciled plus the unmatched count equals the entry count; nothing is written for them.
TestReconcile_ResubmitSameAmountIsNoop: the same body under a new key writes no row and inserts no job and still counts the run as reconciled; a different amount supersedes the reconciled rows (chain estimate, reconciled, reconciled) and the job brings the totals to the new amount.
TestReconcile_ConcurrentSubmissions: two parallel requests for the same run with different amounts both end with 200; every replaced entry has at most one successor; the valid sum equals the amount of the request that committed last; after both jobs the run totals equal the valid sum.
TestReconcile_JobUpdatesRunsAndSessions: after the job runs.version rose by one for changed runs only; agent_sessions.cumulative_cost_micros equals the valid sum of the session, cost_is_estimate is false when all its valid entries are reconciled, agent_sessions.version rose by one for changed sessions only and cost_baseline_micros is unchanged.
TestReconcile_JobIsIdempotent: running the same job again changes no row and no version.
TestReconcile_Validation: the 400 validation_failed cases of notes 3b and 10 with their pointers and codes; nothing written, no job.
TestReconcile_IdempotencyKey: a replay returns the stored answer with Idempotent-Replay: true and no new rows or job; another body under the same key gives 422 idempotency_key_reuse.
TestReconcile_UnknownMemberCanary: a body with an extra member holding a canary gives 400 validation_failed with unknown_member; the canary appears nowhere in the database (scan of note 14), the server logs or the stored idempotency answer.
TestCostEstimates_Read (R-AZ-28): the app with the scope and an owner without a fresh step-up read the estimates of A1 and A2 for one day with paging; personal runs, mixed currencies and runs outside the range are absent; no item has a channel or thread member; estimate_micros stays the original estimate after a reconciliation and reconciled turns true; the allowed reads write no audit entry; an app without the scope gets 403 scope_missing; an agent, Alex's personal token and Sam's session (not an owner) get 403 forbidden with reason: owner_only, each denial audited.

End-to-end (test/reconcile):

TestReconcileApp_EndToEnd (BR-10 end to end, plan row "canary"): the fixture has two days; workspace W1 maps to A1 and A2, W2 to a third agent A4 in mode organization, W3 has no entry in the map; on the first day A1 and A2 have runs with different estimates, one run with estimate 0 and one in another currency, A4 has one run, and A3 (personal) has a run that no workspace can reach; the second day has an amount for W1 but no run. Expected shares are written into the test as literal values computed by hand from the fixture. After the app's pass and the jobs: per day and workspace the reconciled valid entries of the mapped runs equal those values and sum to the fixture amount in micro units; the run with the largest estimate carries the remainder; the zero, the other-currency and the personal runs stay estimates; the app's stderr has not_assigned lines for W3 (no_mapping) and for W1 on the second day (no_runs); then the canary scan of note 14 finds nothing and the positive control holds.

Unit and integration (Python, plugins/cost-reconciler/tests, fakes on 127.0.0.1 port 0, no network):

test_distribute.py (BR-10): a table of cases and a seeded random loop: the shares sum to T; the remainder sits on the largest estimate (ties to the smallest run ID); zero estimates and other currencies are left out; no run gives no_runs; only zero estimates give zero_estimates; T 0 gives all shares 0.
test_amounts.py: exact sums of decimal cent strings, the conversion to micro units with half-to-even rounding, invalid strings rejected.
test_config.py: every configuration error of note 11c names the key and never the value; defaults from 10 17; http accepted only for loopback.
test_secretref.py: file: with modes 0600 and 0400 accepted, 0644 rejected (SEC-SEC-1); env: accepted; other schemes rejected; repr, str and formatting of a secret show only the redaction mark.
test_costapi.py: request path, window of complete UTC days, grouping and daily buckets, paging to the end, the interval of reconciler.min_request_interval between requests with a fake clock, no redirect followed (the fake's redirect target receives nothing), an error status logged without the body, a malformed page failing the pass before any submission.
test_server_client.py: bearer token, a fresh UUIDv7 Idempotency-Key per submission (version and variant bits, order), the handling of each status class of note 11i.
test_pass.py: one submission per bucket with entries, none for buckets without entries, the not_assigned, currency_mismatch and unmatched_runs log lines, exit codes of --once.
test_canary.py: with a canary admin key the fake cost API receives it in the authentication header, and it appears in no request to the fake server (method, path, query, headers, body), no log line, no exception text (also on 401 and 500 from the cost API) and no repr of the configuration.
test_manifest.py: manifest.json has exactly the scope costs:reconcile, version equal to VERSION, no events, delivery, commands, components or grants.
test_doc_gate.py: ruff check with the project's configuration on a temporary module with an undocumented public function reports a D finding and exits non-zero (23 "Documentation in code": every gate has a test).

Test IDs of the plan row and the chapters:

BR-10 (10 15): "Reconciliation distributes the daily amount over estimates: Sum equals actual amount, rounding remainder at the largest entry, estimates replaced": test_distribute.py, TestReconcile_BR10_EstimatesSuperseded, TestReconcileApp_EndToEnd.
Canary "the server stores no vendor key": TestReconcileApp_EndToEnd, test_canary.py, TestReconcile_UnknownMemberCanary, TestServerConfig_HasNoReconcilerKeys, and make semgrep with bothy-go-no-model-credentials.
"both paths apply the entries through the same job billing.reconcile": TestReconcile_BothPathsUseSameJob.
"cost.reconcile requires step-up, an app token is rejected there (R-AZ-03)": TestReconcileCosts_StepUpRequired, TestReconcileCosts_TokensRejected, TestCostReconcileRequirement.

Commands that must pass, from the repository root:

cd services/server && go test -race -count=1 ./internal/billing/... ./internal/runs/... ./internal/agents/... ./internal/authz/... ./internal/wire/... ./test/reconcile/...
cd plugins/cost-reconciler && uv run --locked pytest
make fmt-check, make lint, make typecheck
make codegen-check, make arch, make api-lint, make api-checks, make register-check, make config-doc-check
make semgrep, make gitleaks, make licenses, make docs (it checks that the output of make config-reference is committed)
make check

Acceptance criteria¶

Definition of done¶

Code and tests as specified; doc comment on every public symbol in Go and a docstring on every public module, class and function in Python (purpose, parameters, return value, errors, example where useful); comments explain why.
READMEs of billing, runs, agents, test/reconcile, plugins/cost-reconciler and the row in plugins/README.md updated (port, event, job, lock order, the query exceptions, the app's setup, security model and limits).
Rulings of note 17 recorded in docs/DECISIONS.md section 10 (one row each with date, decision, reason and place).
services/server/CHANGELOG.md entry under 'Unreleased'; plugins/cost-reconciler/CHANGELOG.md with its first version.
An ADR only if you take an architecture decision beyond this brief (MADR 4 as in docs/adr/0000-template-madr-4.md, headings and text in English, next free number, make adr-index).
make check green.
An independent code review over the diff, then the red team gate (docs/quality/red-team-gate.md, verdict JA or NEIN), with attention to T-PL-01, T-TK-02 and RK-17: can any path put the admin key into the server or its logs, can an app token reach cost.reconcile, can a submission change or delete a non-reconciled entry or reconcile a personal run, can the app send the key to another host; on NEIN fix with a test per finding and have the delta checked again. Record both verdicts.
Land the change as AGENTS.md ("How work lands on main") describes: a pull request from this branch, squash-merged only when review, red team and make check, ci-ok and the review threads are green; then git worktree remove.

Report¶

Write to {{REPORT_PATH}}:

First line exactly Verdict: DONE or Verdict: BLOCKED <reason>.
Commits (SHA and subject), changed files with line counts, test results with counts (the red run before the change and the green run after), the summary of make check.
Review and red team verdicts and the findings fixed.
What is not proven.
Follow-ups that could not be done in this scope, each with its reason, and spec gaps found.

Then write {{DONE_PATH}} with exit=0 sha=<sha> or exit=1 <reason>.