08 Nov 2023 AI Engineering 5 min read

Evaluating LLM Apps Beyond Vibes

Why demos and gut feeling are not an evaluation strategy for LLM applications. We cover golden test sets, model-graded evaluation with GPT-4 as judge, the position and verbosity biases quantified by Zheng et al. in 2023, and a layered evaluation stack that runs in CI — including what each technique does not measure.

The Problem With Shipping on Vibes

An LLM feature passes review today the way demos always have: someone types three prompts, the answers look good, the ticket closes. We call this evaluation by vibes. It is not evaluation. LLM outputs are sampled from a probability distribution; a handful of readings tells you almost nothing about the failure rate. Classical unit tests do not transfer either, because there is no single correct string to assert against.

A useful evaluation answers three questions. Did this prompt change make the system better or worse? By how much? On which cases? Without numbers, every prompt edit, model upgrade and parameter change is a blind bet. This article summarizes what worked for us in 2023: golden test sets, model-graded evaluation and a clear view of where both stop working.

A change lands — prompt, model or retrieval. 1/4

Why Demos Mislead

A demo is a selected sample. The presenter has typically tried many inputs and shows the ones that worked, so the audience sees survivorship, not the base rate. A feature that succeeds on 80% of requests demos flawlessly and still fails every fifth user. With sampling temperature above zero, even the shown input is not guaranteed to reproduce.

Hosted models add silent drift. API providers update models without changing your code; behavior on your inputs can shift between deployments. Without a fixed test set run before and after, you cannot distinguish a model regression from a prompt bug. A demo proves existence — the system can produce a good answer — never reliability.

Golden Test Sets as Ground Truth

A golden test set is a versioned collection of representative inputs with expected results, stored next to the code and run on every change. Start small: 50 to 200 cases drawn from real usage, plus known edge cases and every production failure you have triaged. Each case carries an assertion: exact match, substring, regular expression, JSON-schema validity or a semantic check.

Be honest about the limits. A golden set measures regression against a frozen sample, not quality on the live distribution. It does not cover inputs you have not imagined, and it says nothing about subjective quality — tone, helpfulness, style — unless you define a grading function for it. That grading function is where LLM-as-a-judge enters.

Maintenance is part of the definition. When production surfaces a new failure, it becomes a case; when the product changes, obsolete cases are removed, not silently skipped. A golden set that is not curated decays into a green dashboard nobody trusts. We schedule the review with the sprint, not with the incident.

What an LLM Judge Actually Measures

Zheng et al. published "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" in June 2023 (arXiv:2306.05685). MT-Bench contains 80 multi-turn questions across eight categories, from writing to math. Chatbot Arena, launched in May 2023, collected roughly 30,000 crowdsourced pairwise votes in its first month. The core result: a GPT-4 judge agrees with human experts in over 80% of cases — 85% on non-tie MT-Bench votes, against 81% agreement between the humans themselves.

Read that number precisely. The judge approximates human preference, not factual correctness. It disagrees with humans about as often as humans disagree with each other, which makes it a usable proxy — and no more than that. Pairwise comparison is more sensitive; single-answer grading against a rubric is cheaper and scales better.

Known Biases of LLM Judges

The same paper quantifies systematic failure modes. All tested judges preferred answers by position, most favoring the first slot; only GPT-4 stayed consistent in more than 60% of position-swap tests on near-identical answers. A "repetitive list" attack — padding an answer with rephrased duplicates — fooled every judge to some degree. On math questions the default judge prompt approved wrong answers in 14 of 20 trials; reference-guided grading cut that to 3.

Self-enhancement bias deserves a careful reading: Zheng et al. observed GPT-4 favoring its own answers with a 10% higher win rate but state that the data is insufficient to prove the effect. Treat it as a design constraint anyway — evaluating a model with itself as the sole judge invites exactly this doubt.

Bias	Evidence in Zheng et al. 2023	Mitigation
Position bias	Only GPT-4 judged consistently in more than 60% of swap tests	Evaluate both orders; count only consistent verdicts
Verbosity bias	Repetitive-list attack fooled all judges; GPT-4 resisted best	Penalize length; grade single answers against a rubric
Self-enhancement bias	GPT-4 showed a 10% higher win rate for its own answers (not conclusive)	Judge with a model family not under test
Limited math grading	14 of 20 wrong answers approved with the default prompt	Reference-guided grading (3 of 20 failures)

An Evaluation Stack for Production

In practice we layer checks by cost. First deterministic assertions: parseable JSON, schema validity, required fields, banned strings. Then cheap semantic checks such as embedding similarity to a reference. Only then an LLM judge — with swapped positions, a rubric and a reference answer where one exists — on the cases cheap checks cannot decide. Human review is reserved for judge-human disagreements and a random sample.

Tooling for this existed by late 2023: OpenAI open-sourced its Evals framework in March 2023, EleutherAI's lm-evaluation-harness covers academic benchmarks, and promptfoo and LangSmith target application-level tests. None of them removes the hard work: writing cases and rubrics that encode what "good" means for your product. Budget GPT-4 judging for samples, not full traffic; it is the most expensive assertion you will run.

This stack is offline. It does not observe live traffic, capture user feedback or detect distribution shift after release. Those need logging, sampled human review and feedback signals in the product itself. Offline evals tell you whether a change is safe to ship; only production tells you whether it works.

Outlook From November 2023

Two days ago OpenAI announced GPT-4 Turbo with a 128K context window and lower prices. Cheaper strong models make judge-based evaluation viable at larger scale, and longer contexts allow judging whole conversations instead of single turns. We expect fine-tuned open-weight judge models to narrow the gap to GPT-4 within the next year, reducing the dependency on a single vendor for the measuring instrument.

Our prediction: by the end of 2024, eval suites will be as unremarkable in LLM projects as unit tests are in software projects, and CI will fail builds on eval regressions. Teams that maintain golden sets today will switch models in days, not months, when better ones arrive. At Blue IT Systems we now treat the evaluation suite as a deliverable of every LLM project — the demo is marketing; the eval set is engineering.