Decision Doc: The Experiment Brief That Replaced Our Offline Evals

Most portfolios describe decisions. This series publishes the documents behind them. What follows is a sanitized version of a real experiment brief from my current role. Names and internal system references are removed. The numbers, constraints, and outcome are real.

Context for readers. We ran a document-automation portfolio in a regulated tax domain. Our offline evaluation suite scored every release candidate against a fixed labeled set, and releases shipped when the score improved. The brief below was written the week we discovered that this score barely predicted anything that mattered.

The brief, as written

Problem. Offline eval scores show near-zero correlation with case outcomes. Two releases that improved the offline score produced no measurable improvement downstream, and reviewer escalations rose after one of them. We are optimizing a number that does not represent the product.

Hypothesis. Scoring release candidates against replayed production traffic, rather than a curated labeled set, will rank candidates in the same order as their real downstream effect. If true, replay becomes the release gate. If false, we learn our production variance is too high to measure this way, which is also worth knowing.

Design.

1. Capture twelve weeks of production documents with their final human-approved outcomes. This is the replay corpus. It is refreshed quarterly so it cannot fossilize the way the labeled set did.
2. Run every release candidate against the full corpus in shadow mode. The candidate sees real inputs. No output reaches a case.
3. Score agreement against the human-approved outcome, not against the original label. Segment by document type, because aggregate scores hide exactly the failures that hurt us.
4. Gate: a candidate ships only when replay agreement is at or above the current system on every major segment, not on average.

What would change my mind. If replay agreement and downstream amendment rates diverge over two release cycles, replay is also a proxy and we go back to instrumenting outcomes directly.

Cost. One engineer-week to build the replay harness against existing capture infrastructure, then roughly one compute-hour per candidate. Cheap relative to one bad release.

What happened

The first replay run caught a regression that the old suite had scored as an improvement. The candidate did better on the curated set and worse on three of the five highest-volume document segments. We held the release.

The same harness later let us replay 30,000 historical documents while testing a cheaper model mix and canary 5 percent of traffic on the way out. Model quality improved 34 percent and cost fell 41 percent with no rollback. The replay corpus, not the model swap, is what made that migration boring.

The larger lesson sits in the hypothesis section. An eval suite is a product decision about what you are willing to be wrong about. Ours had quietly decided that a curated yesterday mattered more than a messy today, and no one had written that down where it could be challenged.

If you are building your own gate, start with the sentence that took me longest to write: what would change my mind. An experiment brief without that sentence is a press release.

You can play this gate yourself: the Release Gatekeeper walkthrough puts you in front of the same five decisions with the same evidence. Related: how we turned this gate into a scoring rubric an LLM could apply is covered in The Art of the Ask.