A client’s product description generator ran fine for four months. Then someone tightened the system prompt to discourage superlatives, shipped it on a Thursday, and by Monday the thing was emitting descriptions that silently dropped the SKU and dimensions from about one in six outputs. Nobody caught it in review because the prose read beautifully. The retailer’s merchandising team caught it.

That’s the whole argument for an eval harness in one paragraph. Prompt and model changes don’t fail loudly like a type error. They fail on a subset of inputs you didn’t look at, in a direction you weren’t watching, and the output still reads like confident English.

You don’t need a platform for this. You need about 200 lines of code, 40 test cases, and the discipline to run it before you merge. Here’s the version we actually use on client work, including the parts that took two or three attempts to get right.

Why eyeballing output stops working at case 20

Manual review has a specific failure curve. With five outputs you notice everything. With twenty you notice tone and skim the facts. With a hundred you’re pattern-matching on the first two lines and clicking approve. The cases that break production are almost never the ones you read closely: they’re the weird input, the empty field, the customer name with an apostrophe, the 4,000 character source text that pushes the important instruction out of the model’s attention.

The second problem is regression blindness. You improve one thing and degrade another, and without a baseline you have no way to see the trade. Tighten a prompt to stop hallucinated features and you may also kill the call-to-action. We’ve done exactly that. Twice.

Treat ai output quality as a test surface rather than a review task. The goal isn’t a perfect score. The goal is knowing, within five minutes of a change, whether you just made things worse and where.

The three layers of LLM evaluation worth building

Layered, cheapest first. Each layer only runs if the previous one is worth running.

1. Structural: does it parse? JSON schema valid, required keys present, enum values legal, no markdown fences where you asked for raw JSON. Free, instant, binary.
2. Assertional: does it contain what it must and omit what it must not? SKU present, price matches input, no mention of competitors, under 160 characters, no banned phrases. Still free. Still deterministic.
3. Judged: is it any good? Tone, faithfulness to source, helpfulness. Costs money, introduces variance, needs calibration. Use it last and on fewer cases.

Most teams start at layer three because it feels like the real evaluation. That’s backwards. On a product description task we shipped last year, layers one and two caught around 75 percent of the failures that mattered, including every single one that would have reached a customer unnoticed.

Writing cases that catch real regressions

A test case is an input, a set of expectations, and a stable id. Keep them in plain files under version control, not in a SaaS dashboard, so a prompt change and its test changes land in the same diff.

// evals/cases/product-description.js
export default [
  {
    id: "pd-001-basic",
    input: { sku: "CNV-4421", name: "Oak Side Table", dims: "45x45x50cm", price: 189 },
    assert: [
      { type: "contains", value: "CNV-4421" },
      { type: "contains", value: "45x45x50" },
      { type: "not-contains-any", value: ["best-in-class", "revolutionary", "game changer"] },
      { type: "max-words", value: 90 }
    ]
  },
  {
    // Regression: shipped 2026-01-14, dropped SKU when dims field was empty
    id: "pd-014-empty-dims",
    input: { sku: "CNV-9007", name: "Linen Cushion", dims: "", price: 29 },
    assert: [
      { type: "contains", value: "CNV-9007" },
      { type: "not-contains", value: "undefined" },
      { type: "judge", rubric: "faithfulness" }
    ]
  }
]

The comment on pd-014 matters more than the assertions. Every case should be traceable to a reason it exists. When a case starts failing two quarters later and nobody remembers why it was written, it gets deleted, and then the bug comes back.

Where the cases come from

• Production failures. Highest value by a wide margin. Freeze the exact input that broke.
• Edge shapes. Empty string, missing key, 10x expected length, non-Latin script, emoji in a user name, HTML in a text field.
• Adversarial inputs. Prompt injection in the data you interpolate. If you’re building anything agentic, this overlaps heavily with the guardrail work covered in our piece on AI agents architecture and failure modes.
• Happy paths. A handful, to confirm you haven’t broken the boring 90 percent.

Thirty to sixty cases is a working suite. Below 30 the signal is too noisy to act on; a single flip moves your score by three points. Above 200 you’ll stop running it.

The runner: 80 lines that do the work

No framework needed. A loop, a dispatch table of assertion types, and a JSON report.

// evals/run.js
import cases from "./cases/product-description.js";
import { generate } from "../src/generate.js";
import { judge } from "./judge.js";

const CHECKS = {
  contains: (out, a) => out.includes(a.value),
  "not-contains": (out, a) => !out.includes(a.value),
  "not-contains-any": (out, a) =>
    !a.value.some(v => out.toLowerCase().includes(v.toLowerCase())),
  "max-words": (out, a) => out.trim().split(/\s+/).length  { try { JSON.parse(out); return true } catch { return false } }
};

const N = Number(process.env.EVAL_N ?? 1); // repeat runs to expose variance

const results = [];
for (const c of cases) {
  for (let run = 0; run < N; run++) {
    const out = await generate(c.input);
    const checks = [];
    for (const a of c.assert) {
      if (a.type === "judge") {
        const { score, reason } = await judge(a.rubric, c.input, out);
        checks.push({ type: judge:${a.rubric}, pass: score >= 4, score, reason });
      } else {
        checks.push({ type: a.type, pass: CHECKS[a.type](out, a) });
      }
    }
    results.push({ id: c.id, run, pass: checks.every(k => k.pass), checks, out });
  }
}

const byCase = new Map();
for (const r of results) {
  const e = byCase.get(r.id) ?? { passes: 0, total: 0 };
  e.passes += r.pass ? 1 : 0; e.total++; byCase.set(r.id, e);
}
const rate = [...byCase.values()].reduce((s, e) => s + e.passes / e.total, 0) / byCase.size;

console.table([...byCase].map(([id, e]) => ({ id, pass: ${e.passes}/${e.total} })));
console.log(Weighted pass rate: ${(rate * 100).toFixed(1)}%);
await Bun.write("evals/last-run.json", JSON.stringify(results, null, 2)); // or fs/promises
process.exit(rate >= Number(process.env.EVAL_THRESHOLD ?? 0.9) ? 0 : 1);

Two things in there are deliberate. Writing the full output to last-run.json means you can diff two runs and read what actually changed, which is the single most useful debugging artefact in this whole setup. The weighted pass rate averages per case rather than per run, so a flaky case can’t dominate just because it ran five times.

LLM as judge, without fooling yourself

A judge model is useful and also the easiest place to generate confident nonsense. Known biases, all reproducible: judges prefer longer answers, prefer whichever response is shown first in pairwise comparison, and rate their own family’s output higher. A 1 to 10 scale produces clustering at 7 and 8 with no discriminating power.

Use a 1 to 5 integer scale, demand a reason before the score, force structured output, and keep each rubric to one dimension.

// evals/judge.js
const RUBRICS = {
  faithfulness: `Score how well the description sticks to the supplied facts.
5 = every claim is supported by the input, nothing invented.
4 = all facts correct, minor unsupported flourish.
3 = one unsupported specific claim (material, origin, feature).
2 = multiple invented specifics.
1 = contradicts the input.`,
  tone: `Score adherence to a plain, concrete retail voice.
5 = concrete, no superlatives, no hype adjectives.
3 = one or two hype phrases.
1 = marketing-speak throughout.`
};

export async function judge(rubric, input, output) {
  const res = await openai.chat.completions.create({
    model: "gpt-4.1-mini",
    temperature: 0,
    messages: [
      { role: "system", content: `You are a strict evaluator. ${RUBRICS[rubric]}
Give your reasoning first, then the integer score. Do not reward length.` },
      { role: "user", content: INPUT FACTS:\n${JSON.stringify(input)}\n\nOUTPUT:\n${output} }
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "verdict", strict: true,
        schema: {
          type: "object",
          // reason before score: the model's own text conditions the number
          properties: { reason: { type: "string" }, score: { type: "integer", enum: [1,2,3,4,5] } },
          required: ["reason", "score"], additionalProperties: false
        }
      }
    }
  });
  return JSON.parse(res.choices[0].message.content);
}

Calibrate before you trust it

Score 30 to 50 outputs yourself, by hand, into a spreadsheet. Run the judge on the same set. Compare. If exact agreement is below about 70 percent, or your scores and the judge’s disagree by two or more points on a handful of cases, your rubric is ambiguous rather than your model being bad. Rewrite the rubric with the disagreements as anchor examples and go again. Cohen’s kappa is the proper measure here if you want one number; anything in the 0.6 to 0.8 range is respectable for subjective text quality.

Recalibrate when you change judge model. A judge upgrade can shift your mean score by half a point with no change to the system under test. That’s exactly the kind of phantom regression that destroys trust in the suite.

Running it in CI without burning the budget

Full suite on every push is the mistake. Fifty cases times three judged assertions times five repeats is 750 calls, and at pull request frequency that’s real money and a four minute wait people learn to ignore.

What works: a 12 case smoke set on every PR touching prompts or generation code, the full suite nightly and on release branches.

# .github/workflows/evals.yml (fragment)
on:
  pull_request:
    paths: ['src/prompts/', 'src/generate.js', 'evals/']
  schedule:

cron: '0 3   *'

smoke on PRs, full nightly

run: EVALSUITE=smoke EVALN=1 EVAL_THRESHOLD=1.0 node evals/run.js
  if: github.eventname == 'pullrequest'

run: EVALSUITE=full EVALN=5 EVAL_THRESHOLD=0.92 node evals/run.js
  if: github.event_name == 'schedule'

Note the different thresholds. The smoke set is small and hand-picked, so it should be 100 percent, every time, no exceptions. The full suite has a tolerance because at n=5 across 50 cases you’ll always have some genuine stochastic variance. Caching helps too: hash the prompt template plus the input, and skip generation if neither changed since the last green run. On a suite that’s mostly stable, that cuts spend by more than half.

Reading the number without lying to yourself

A single pass rate is a smoke alarm, not a diagnosis. When it drops, the useful questions are narrow: which cases flipped, in which assertion category, and is the output materially different or just differently phrased?

Three rules we hold to:

• Never tune the rubric to make a failing build pass. If you decide a case was wrong, say so in the commit message and delete it properly.
• Track cost and latency in the same report. An output quality win that triples token spend is a decision, not an improvement. We log prompt tokens, completion tokens and wall time per case; it’s three extra fields and it has killed at least two “better” prompts on economics alone.
• Keep a frozen baseline. Pin the model version in the harness config. Providers deprecate and silently reroute snapshots, and if you’re weighing self-hosted alternatives for version stability, the trade-offs are laid out in open source models in production.

When this is the wrong investment

If a human reads and approves every single output before it reaches anyone, you don’t need a harness yet. You need a decent review queue and a log. The harness earns its keep when output goes out unattended, when volume makes real review impossible, or when more than one person edits the prompts.

It’s also the wrong tool for genuinely open-ended creative generation where there is no faithfulness criterion. Judging a marketing headline on a 1 to 5 scale produces a number that correlates with nothing you care about. Score the constraints you can define (length, banned terms, brand vocabulary, factual claims) and leave the taste judgement to a person. Related: if any of that output is published as-is, the disclosure question is worth settling early, which we covered in the ethics and disclosure of AI-assisted content.

And if you’d rather not maintain a runner at all, promptfoo gives you YAML-defined cases and a decent diff view out of the box, DeepEval plugs into pytest, and Braintrust or Langfuse add tracing and dataset management. All reasonable. Build your own first anyway, even if you throw it away: writing the assertions by hand is how you discover what quality actually means for your task.