Fifty labelled examples in a JSONL file will do more for your LLM app than any eval platform you're evaluating instead of building one. The order of operations: golden set first, code assertions before model-graded ones, calibrate any LLM judge against your own human labels before trusting a single score from it, then gate CI on regressions. Teams invert this — standing up dashboards before they have ten labelled examples — and end up with precise measurements of nothing.

The golden set: 50 examples from production

Mine real traces, not imagined inputs. Pull 50–100 production requests (or beta traffic, or your own dogfooding) and label what a good output looks like for each. Two selection rules make the set earn its keep:

  • Cover your failure taxonomy. List the ways your app fails — hallucinated fields, missed entities, wrong tone, over-refusal — and make sure each mode has cases. A set of only easy inputs tests nothing.
  • Include the traps. Ambiguous inputs, adversarial phrasing, inputs where the correct answer is "I don't have enough information". These are where regressions hide.

Format is deliberately boring:

{"id": "t-041", "input": "...", "expected": {"component": "auth", "severity": "high"},
 "tags": ["ambiguous-severity"], "source": "trace_8f3a", "labelled_by": "am", "date": "2026-05-11"}

Then adopt the discipline that makes the set appreciate in value: every production incident becomes a case. Same rule as regression tests, because that's what this is. Six months in, your golden set is a catalogue of every way your system has actually failed, and nothing re-breaks silently.

The assertion ladder: code first, judge last

Grade with the cheapest assertion that catches the failure. In order:

  1. Deterministic checks — output parses, schema validates, required fields present, length bounds, no PII patterns, cited document ID actually appears in the retrieved set. Free, instant, zero variance. In most pipelines these catch the majority of real regressions.
  2. Reference comparison — exact match for classification, contains / regex for extraction, embedding similarity above a threshold for near-paraphrase tasks.
  3. LLM-as-judge — only for genuinely open-ended qualities (helpfulness, faithfulness to context, tone), and only calibrated.

The ladder matters because judge scores are the most expensive and least trustworthy rung. If "did it output valid JSON with a real severity label" covers 70% of what you care about, you get 70% of your eval for free and reserve the judge budget for the questions that need it.

Calibrating the judge, not vibing with it

An uncalibrated LLM judge is vibes with extra latency. Before its scores gate anything: take 100 outputs, label them yourself, run the judge on the same 100, and measure agreement. Aim for ~85%+ raw agreement (Cohen's kappa around 0.7) before the judge's opinion counts; below that, fix the rubric, not the threshold.

Design around the known biases, documented since MT-Bench and still present in current models:

  • Position bias — in pairwise comparisons, judges favour the first answer. Run both orderings; a result that flips is a tie.
  • Verbosity bias — longer answers score higher independent of quality. State explicitly in the rubric that length is not quality.
  • Self-preference — models rate their own family's style higher. Judge with a different family than the generator when you can.
  • Scale incontinence — a 1–10 score is noise; the same output gets a 6 or an 8 on different runs. Decompose into 2–4 binary questions ("Does the answer use only facts from the context: yes/no") and sum. Binary questions are also what you can calibrate against.

Re-run the calibration check quarterly and whenever you change the judge model — judge drift is real and invisible without it.

CI gating

The eval runs like any other test suite: on every PR touching prompts, retrieval, or model config, compare against the baseline and block on regression.

# .github/workflows/eval.yml (excerpt)
- run: python evals/run.py --suite golden --out results.json
- run: python evals/gate.py results.json --baseline main --max-drop 0.02

Practical settings that keep it usable: run cases concurrently so the suite finishes in under five minutes; pin the judge model version; allow a small tolerance (1–2 points) on judge-graded metrics to absorb variance, but zero tolerance on deterministic checks; and print the per-case diff on failure so the author sees which inputs regressed, not just a number. Cost at this scale is irrelevant — 100 cases with a small judge model is a few cents per run. This is also where prompt changes get their merge/no-merge answer, which is the entire point: the eval converts prompt review from opinion to evidence.

Drift detection: the eval, pointed at production

Offline evals catch changes you made; drift detection catches changes that happened to you — model provider updates, input distribution shifts, a dependency "improving" retrieval. The minimal setup: nightly, sample 50–100 production traces, run the deterministic checks plus the calibrated judge, and write the scores to a time series. Alert on a sustained drop over ~3 days rather than single-night noise.

Watch the free proxy metrics alongside: validation-retry rates, tool-call repair rates, refusal rates, user regenerate/thumbs-down rates. They're noisy individually, but a judge-score drop that coincides with a retry spike is a real incident. Wire the samples through the same tracing infrastructure you already run — a trace store like Langfuse can hold scores next to the traces they grade, which is exactly where you want them during an incident review.

What I'd do

Today: 50 cases from real traffic into a JSONL file, deterministic assertions, a script that prints pass/fail. This week: wire it into CI against a baseline. This month: label 100 outputs, calibrate a judge on the open-ended half of your quality bar, add the nightly production sample. Resist the platform purchase until the file outgrows you — the JSONL file plus 200 lines of Python is the eval system; everything else is a nicer UI for it.