There's a moment in every AI project where someone asks, “Is it better now?” — and everyone looks at the person who tried three prompts this morning. That moment is the fork in the road. One path leads to production; the other leads to a demo that dies quietly in a pilot.

The difference is an evaluation harness. Here's the version of that argument that has actually convinced CFOs and heads of engineering — and the concrete shape of the thing we build first on every AI engagement.

Why the harness comes before the feature

An AI feature without evals is a feature whose quality you learn about from customers. Every change — a new model version, a reworded prompt, a retrieval tweak — either helps or hurts, and without a harness you find out anecdotally, weeks later, through support tickets.

We build the harness first because it changes three conversations:

* With engineering: “does this change help?” becomes a CI check, not a debate.

* With finance: cost per request becomes a tracked number with a budget, not a surprise on the invoice.

* With compliance: “what will the model say?” becomes a tested distribution, not a hope.

On a fintech reconciliation agent we took to production, this was the whole ballgame: the internal team had a working prototype, but evals were manual, cost was unbounded, and finance wouldn't sign off. The rebuild put an eval harness in CI with 92% regression coverage — hallucinations went from ~7% at handover to under 0.4% at ship, and the feature went live in eight weeks.

The four parts, concretely

1. A golden set. 100–500 real inputs with agreed-correct outputs, drawn from actual usage — not synthetic examples. It grows every time production surprises you: every bad output becomes a test case, the way every incident becomes a regression test.

2. Offline evals in CI. Every prompt change, model swap, or retrieval adjustment runs against the golden set before merge. Scoring is layered: exact-match where the answer is checkable, model-graded where it's judgment, human review for the sliver that's genuinely ambiguous.

3. Prompt versioning. Prompts are code. They live in the repo, ship through review, and roll back like any other artifact. “Who changed the prompt?” should have a git blame answer.

4. A cost budget per request. Model tiers, caching layers, and per-tenant budgets — with a unit-cost dashboard from week one. The feature that delights at $0.002 per request dies at $0.04; you want to see that curve long before your CFO does.

What it costs — and what it returns

The harness is typically the first two weeks of an AI engagement. Teams sometimes push back: “we want the feature, not the scaffolding.” Our answer is blunt: without the scaffolding you don't have a feature, you have a demo with a countdown timer.

The return shows up fast: model upgrades become an afternoon's work instead of a re-testing project, cost cuts become safe (eval-guided prompt trims cut one client's cost per request by 64%), and the answer to “is it better now?” becomes a number everyone can see.

The test for your own team

If a new model version dropped tomorrow, could you say — with evidence, by end of day — whether switching helps or hurts your feature? If yes, you have a harness. If no, that's the first thing to build, and it's exactly where we start in our AI Engineering practice.