feat(alexandria): provider discovery through Search and billed execution through Scrape - #4621

Summary

Alexandria provider tools on the existing Search and Scrape surfaces, following the frozen Firecrawl/Exchange interface.

• POST /v2/search with sources: ["alexandria"] (optionally domainTools) returns tool contracts in data.tools, up to limit per discovery source. Discovery is free and never counts toward creditsUsed or the keyless reservation; web results bill as before.
• POST /v2/scrape with a url accepts domainTools: true (default off) and adds data.tools matched to the scraped page's domain, same shape as Search, free, only for teams with Exchange access.
• POST /v2/scrape with an alexandria body executes one to ten provider calls and returns data.alexandria with data.creditsCost. /exchange/retrieve shares the same path; its single-call shape relays a provider error with the Exchange's status and code.

Billing

Execution runs inline in the request on existing rails: authorize through Exchange /v1/provider-terms/requirements (404 for unknown providers, so nothing unlisted reaches a quote or execution) against the organization's organizationDataSourceAccess flags, quote, reserve with Autumn (lockCredits, lock alexandria_<chargeId>), execute /v1/retrieve with budget and deadline headers, settle the receipt with finalizeCreditsLock, enqueue the ledger write on the billing queue with a deterministic job id, and report to /v1/usage-events/billing. Paid requests fail closed when authorization, quote, reservation, or USE_DB_AUTHENTICATION is unavailable. No new queue, worker, or migration.

The charge id is sha256(teamId, x-request-id). One Redis record per charge id (7 days) gives: replay of a completed request with its original scrape_id and no second execution, 409 duplicate_request for a different payload, 409 request_in_flight while running, release of the record on any pre-execution refusal so the id can retry, and 503 request_unresolved held for reconciliation when the Exchange may have executed (5xx, timeout, malformed or over-budget receipt, crash after the point of no return). The Exchange has no request-id idempotency, so unresolved requests are never re-executed automatically. A failed ledger commit is refunded at Autumn on the direct route; on the firebill route the durable charge stands pending reconciliation.

Details in apps/api/src/services/alexandria/README.md.

Verification

• Unit: 47 tests across services/alexandria (retrieve, access), the controller, and search.
• Typecheck, knip, prettier clean.
• Live against firecrawl-local (API + TLS Exchange snapshot with the provider-terms, quote, and skills-resolve routes + Exchange Postgres): 17 of 17 smoke checks pass, plus a fetch-harness check that a URL scrape with domainTools: true returns 21 World Bank tools with domain provenance and none without the flag, including one usage row per execution, confirmed billing status on the Exchange, same-id and cross-route replay with a stable scrape_id, concurrent same-id dedupe, 409 codes, unknown-provider refusal before any quote, seeded unresolved and in-flight records, and fail-closed paid execution without Autumn.

Not yet validated

Autumn is not configured locally, so every paid path stops at the reservation. The confirm, the release after an Exchange 4xx, and the ledger enqueue are mock-tested only. One paid call against the Autumn sandbox is the remaining check before merge.

Exchange dependencies

/v1/retrieve/quote, x-exchange-max-credits, and /v1/skills/resolve are on Exchange codex/contextual-provider-tools; /v1/provider-terms/requirements is in the Exchange provider-terms worktree (uncommitted there as of 2026-09-11). Neither is on Exchange main. /v1/usage-events/billing is on main and needs a non-blank EXCHANGE_INTERNAL_SECRET.

Summary by cubic

Adds Alexandria provider tools to Search and Scrape: the alexandria source / domainTools on Search, and domainTools on URL scrape, attach free tool contracts to data.tools, while Scrape's alexandria body executes 1–10 paid provider calls and returns data.alexandria. All are gated by the exchangeRetrieve flag; domainTools without access (or under ZDR) returns 403 on both surfaces, and the legacy /exchange/retrieve route shares the Scrape handler. The previously used exchange body key is rejected as unrecognized.

Billing and idempotency

• Provider execution runs inline on existing billing rails; no new queue, worker, or migration.
• The charge id is sha256(teamId, x-request-id), so send x-request-id on every paid request or a retry is a new charge.
• A Redis record keyed by that id replays completed requests without re-executing, answers 409 for a different payload or in-flight request, and holds uncertain outcomes as 503 without auto-retry.
• Pre-execution refusals release the record so the same id can retry.
• Paid requests fail closed when authorization, quote, reservation, or USE_DB_AUTHENTICATION is unavailable.
• If the ledger enqueue fails after retries, Exchange usage stays pending for reconciliation; the direct-Autumn charge is refunded and firebill charges stand.

Rollout

• Requires Exchange routes only in the codex/contextual-provider-tools and provider-terms worktrees, not on main.
• Set EXCHANGE_INTERNAL_SECRET (non-blank) for usage-event billing reports.
• Paid settlement is mock-tested only; run one call against the Autumn sandbox before merging.

^{Written for commit 47ef264. Summary will update on new commits.}