Eval-Driven Development for LLM Systems
Demos prove an LLM system can work once; evals prove it works reliably. This article defines golden test sets, the four standard RAGAS metrics for RAG pipelines, LLM-as-judge with its documented biases and verified mitigations, offline versus online evaluation, and CI regression gates — plus trajectory and tool-call evaluation for agents, with numbers from MT-Bench, τ-bench, and the Berkeley Function-Calling Leaderboard.
Why the Demo Misleads
A demo shows one sampled path through a non-deterministic system. Large language models produce different outputs for identical inputs, so a demo proves existence — the system can produce this answer — not reliability — the system will produce it. Classical unit testing assumes deterministic outputs; that assumption does not hold here. OpenAI's evaluation guidance names the resulting anti-pattern directly: "vibe-based evals" — shipping because the output seems fine. Evals, meaning structured and repeatable measurements against defined criteria, are the replacement.
The cost of not measuring is concrete. Airbnb's engineering team, which evaluates generative AI at scale, lists three recurring failure modes of eval-free development: false confidence (a generic helpfulness metric scores well while the failure mode users actually hit goes unmeasured), undetected regressions (a prompt change silently degrades a dimension nobody tracks), and wasted effort (pipelines optimized for metrics that do not correlate with outcomes). All three cost more after launch than before.
Golden Test Sets as the Foundation
A golden test set is a curated collection of inputs with human-reviewed reference outputs or labels. It operationalizes what "good" means for one specific application. Airbnb's published recommendation: start with 50–100 examples labeled by subject-matter experts, and include bad examples — discernment cannot be tested against a set that contains only successes. One hard rule: if two experts disagree on a label, resolve the disagreement before automating anything. A judge calibrated against inconsistent labels calibrates to noise.
A golden set is not static. Production failures are the best source of new cases: every meaningful failure becomes a regression test, and over time the set becomes a record of the application's real quality requirements. What a golden set does not do: it cannot detect drift toward traffic it does not contain. That is what online evaluation is for.
The Standard RAG Metric Set
For retrieval-augmented generation, the RAGAS framework (Es et al., 2023) established a de-facto standard of four metrics, each scored 0 to 1. Two grade generation: faithfulness is the fraction of claims in the answer that the retrieved context supports; response relevancy reconstructs questions from the answer (three by default) and measures their mean embedding similarity to the question actually asked. Two grade retrieval: context precision is the mean precision@k over the ranked chunks; context recall measures whether the information required for the reference answer was retrieved at all.
Scope this honestly: faithfulness is not factual correctness. An answer can be perfectly faithful to a wrong or outdated context and still be wrong. All four metrics also use an LLM or embeddings in their own computation, so they inherit judge noise. Treat them as trend indicators over a fixed dataset, not as absolute truths.
| Metric | Grades | Question answered | Typical failure caught |
|---|---|---|---|
| Faithfulness | Generation | Are all claims in the answer supported by the retrieved context? | Hallucination beyond the context |
| Response relevancy | Generation | Does the answer address the question actually asked? | Evasive or off-topic answers |
| Context precision | Retrieval | Are relevant chunks ranked above irrelevant ones? | Noisy ranking |
| Context recall | Retrieval | Was all information needed for the reference answer retrieved? | Missing evidence |
LLM-as-Judge and Its Biases
LLM-as-judge means using a strong model to grade another model's output against a rubric. The validating study is Zheng et al. (2023): on MT-Bench, GPT-4's verdicts agreed with human preferences in over 80% of cases — the same rate at which the human raters agreed with each other. The same paper documents why judges cannot be trusted blindly.
Three biases are well replicated. Position bias: verdicts flip when the order of the compared answers is swapped. Verbosity bias: in Zheng et al.'s "repetitive list" attack, answers padded with rephrased duplicates were rated higher; all tested judges were susceptible to some degree, with GPT-4 resisting markedly better than the others. Self-preference: a 2024 study (arXiv 2410.21819) traces it to perplexity — judges score text that is statistically familiar to them higher than human raters do, regardless of who actually wrote it.
The verified mitigations: swap positions and average both verdicts; require chain-of-thought reasoning before the score; use reference-guided judging, where the judge first produces its own answer — in Zheng et al.'s math-grading test this cut judging failures from 14/20 to 3/20. Above all, calibrate: measure agreement with human labels (Cohen's kappa) on the golden set, target values in the high 80s to 90s, and recalibrate periodically. An uncalibrated judge produces false confidence — worse than no judge at all.
Offline and Online Evaluation
Offline evaluation runs on curated datasets before a change ships; online evaluation runs on sampled production traces afterwards. Each catches what the other misses. Offline gates detect regressions against known cases but go stale as traffic shifts. Online evaluation detects drift, unexpected inputs, and new failure modes, but it has no ground truth, and user-feedback signals are sparse and noisy. Guardrails are the synchronous special case of online evaluation: checks in the request path that block an output before it reaches the user.
The two modes form a loop. Offline experiments validate a change before deployment; online sampling finds the cases the dataset did not cover; those cases flow back into the golden set, so the next offline run catches them. Teams that run only one half of the loop measure only half of their system.
Regression Gates in CI
A regression gate runs the eval suite against every proposed change in CI and fails the build when scores drop. The tooling is mature: Langfuse's experiment-action for GitHub Actions, for example, fails the job and posts the scores to the pull request when a run-level score falls below its threshold.
Judge-scored metrics are noisy, so a naive single threshold flaps. Four rules for a gate that holds: 1. Gate deterministic checks hard — schema validity, output format, and tool-call syntax get zero tolerance. 2. Gate judge metrics on deltas against the baseline, using repeated runs or confidence intervals, never a single-run point value. 3. Never let an aggregate hide a critical case: safety- and policy-critical examples must pass individually. 4. Review newly failing examples by hand before overriding a red gate; an override without a diagnosis deletes information.
Evaluating Agents Beyond the Final Answer
An agent's output is a trajectory: a sequence of reasoning steps, tool calls, and state changes. Grading only the final answer is insufficient — a correct answer can mask wrong tool arguments or a broken path, and a plausible transcript can leave the underlying system in the wrong state. Tool-call evaluation therefore decomposes into layers: schema validity, argument correctness (the Berkeley Function-Calling Leaderboard grades this by AST comparison against accepted answers), call ordering, and final state.
τ-bench (Yao et al., 2024) is the reference for state-based agent evaluation: it compares the database at the end of a simulated customer conversation against an annotated goal state, so any trajectory that produces the correct state passes. It also measures reliability with pass^k — the probability that all k independent trials succeed. GPT-4o with function calling reached roughly 61% pass^1 on the retail domain but fell below 25% at pass^8. That gap between one run and eight is exactly what a demo hides.
Include negative cases. BFCL's relevance detection tests inputs where the correct action is no tool call at all. An agent evaluated only on positive cases learns that calling something always beats calling nothing — a lesson it will apply in production.
Outlook: Evals as the Specification
Three developments are visible. First, evals are becoming the specification: a golden set plus evaluator definitions states a system's contract more precisely than a prose requirements document, and model, prompt, and orchestration changes are judged only against that contract. Second, trace standardization — for example OpenTelemetry's GenAI semantic conventions — makes trajectory evals portable across frameworks instead of locked to one vendor. Third, simulation-based evaluation, where agents run against simulated users and sandboxed environments as τ-bench does, is moving from research benchmarks into routine CI.
For our own engineering practice at Blue IT Systems, one operating rule follows from all of the above: a behavior that is not encoded in an eval does not exist as a requirement — and a metric nobody would act on is not worth computing. Everything else is a demo. Teams that internalize this ship slower on day one and faster every week after.
Sources
- Ragas Documentation: Available Metrics (Faithfulness, Response Relevancy, Context Precision, Context Recall)
- Zheng et al. (2023): Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Self-Preference Bias in LLM-as-a-Judge (arXiv 2410.21819)
- Yao et al. (2024): τ-bench — A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Airbnb Engineering: Eval-Driven Development — Lessons from Evaluating GenAI at Scale
- Langfuse: LLM Evaluation — Methods, Best Practices, and a Practical Roadmap
- OpenAI: Evaluation Best Practices
- Berkeley Function-Calling Leaderboard (Gorilla)