Fintech AI Reconciliation

The internal team had a working prototype and no path to production — evals were manual, cost was unbounded, and the finance team wouldn't sign off. We rebuilt the system around a tenant-safe RAG, an eval harness in CI, and a per-request cost budget. It went to production in eight weeks.

The Challenge

The internal team had a working AI prototype and no path to production: evals were manual, cost was unbounded, and the finance team wouldn't sign off. The blocker wasn't the model — it was everything around it: manual evals meant nobody could prove a change helped, and unbounded cost meant finance couldn't sign off even if it did.

Our Approach

A squad of five rebuilt the system around a tenant-safe RAG on LangGraph and Bedrock, an eval harness wired into CI, and a per-request cost budget — making every model change testable and every request accountable. The eval harness went into CI first, making every prompt and model change testable before merge. Cost got an explicit per-request budget enforced by routing and caching, and tenant isolation moved into the retrieval layer itself.

What We Built

A tenant-safe RAG rebuilt on LangGraph and AWS Bedrock; an evaluation harness wired into CI — 92% regression coverage; per-request cost budgets with routing and caching (−64% cost per request); and production hardening: audit trails and graceful failure paths.

Key Decisions

Evals in CI, not on a wiki — every prompt or model change runs the golden set before merge; hallucinations fell from ~7% to under 0.4% because drift literally cannot ship silently. A budget per request — cost stopped being a monthly surprise: routing, caching, and eval-guided prompt cuts run against an explicit per-request budget the finance team signed off on. Tenant isolation in retrieval — financial data demands that one tenant's documents can never inform another's answers; isolation is enforced in the retrieval layer, not by convention.

The Outcome

Production in eight weeks: manual review time down 78%, cost per request down 64%, 92% eval coverage in CI, and hallucinations under 0.4% on the golden set — down from ~7% at handover. What changed wasn't the model — it was accountability: every request budgeted, every change tested, every answer traceable. The prototype's promise, finally deployable.

“They didn't sell us more AI. They made the AI we already had actually deployable.”