LLM Evaluation: Golden Datasets, LLM-as-a-Judge and RAG Metrics

How to evaluate LLM applications: golden datasets, code-based checks, LLM-as-a-judge and its biases, RAG retrieval and faithfulness metrics, release gates.

Intermediate 3 min read · Updated 29 October 2026

LLM evaluation is how you know whether a prompt change, a new model or a retrieval tweak made your application better or worse. Outputs are open-ended text, so classic accuracy rarely applies; instead you combine golden datasets, code-based checks, LLM-as-a-judge graders calibrated against humans, human review, and online signals, and you gate releases on the results. It's the step interviewers most often find missing from AI system designs.

The evaluation loop

Evaluation runs offline before every release and online after it. Production failures flow back into the golden set, so the same mistake can't ship twice.

Building a golden dataset

Start small and real: 50-200 cases drawn from real or realistic user inputs beat thousands of synthetic ones. Grow it from production.
Cover slices: common requests, hard ones, edge cases, adversarial inputs (injection attempts, off-topic, unsafe), each language and customer type that matters.
Store what "good" means: a reference answer, the key facts that must appear, the documents that should be retrieved, or a rubric.
Version it and never train or tune prompts on the held-out portion.
Synthetic expansion: an LLM can generate variations of real cases; review a sample before trusting them.

Kinds of graders

Grader	Checks	Strength	Weakness
Code-based (deterministic)	Valid JSON, schema, contains required fields, exact match, regex, tool called correctly, SQL runs	Cheap, fast, reproducible	Can't judge meaning or tone
Reference-based metrics	Overlap or semantic similarity with a reference answer	Automatic	Penalise correct answers phrased differently
LLM-as-a-judge	Rubric scores: correctness, groundedness, helpfulness, tone; or pairwise preference	Scales to open-ended output	Biased and noisy unless calibrated
Human review	Expert judgment on samples	Ground truth for hard cases	Slow and expensive
Outcome checks	Did the agent's action actually succeed in the system?	Measures what matters	Needs sandboxed environments

Use the cheapest grader that's reliable for each dimension: code checks wherever possible, LLM judges for meaning, humans to calibrate the judges and for the highest-stakes slices.

LLM-as-a-judge done carefully

Specific rubrics: "Score 1 if every factual claim is supported by the provided documents, else 0" beats "rate quality 1-10". Binary or 3-level scales agree with humans more than wide scales.
Calibrate: have humans label a few hundred outputs, measure agreement with the judge per dimension and slice, and only trust dimensions where agreement is high.
Control known biases: judges favour the first answer in pairwise comparisons (randomise order), longer answers (control length), and their own model family's style.
Give the judge the evidence: the retrieved documents and reference facts, not just the answer.
Re-calibrate whenever the judge model, its prompt or the task changes.

Evaluating RAG: retrieval and generation separately

RAG has two places to fail, so measure both: retrieval metrics against labelled relevant passages, and generation metrics for faithfulness to the retrieved context and relevance to the question.

Metric	Question it answers
Context recall (recall@k)	Did retrieval find the passages needed to answer?
Context precision	How much of what was retrieved is relevant?
Faithfulness / groundedness	Is every claim in the answer supported by the retrieved context?
Answer relevance	Does the answer address the question asked?
Correctness	Does it match the reference facts?
Citation accuracy	Do citations point to passages that support the cited claims?

Evaluating agents

Task success verified in a sandboxed system (the refund exists, the test suite passes).
Trajectory checks: right tools, valid arguments, no forbidden actions, no needless steps.
Efficiency: steps, tokens, cost and latency per successful task.
Repeated trials: run each case several times and report the pass rate; one lucky pass proves little.

Release gates and online evaluation

Release gate for prompt v15 vs v14 (golden set: 420 cases, 3 runs each)
  groundedness (judge, calibrated 92% agreement)   0.94 → 0.95   ✓ no regression
  correctness on billing slice                    0.88 → 0.91   ✓
  refusal on unsafe slice                         1.00 → 1.00   ✓ must stay 1.00
  JSON validity (code)                            0.998 → 0.999 ✓
  p95 latency                                     3.1 s → 3.0 s ✓
  cost per conversation                           $0.021 → $0.024  ⚠ +14%, review
→ ship to 5% canary; watch thumbs-down rate, escalations and cost per resolution

Online signals: explicit feedback (thumbs, ratings), implicit (copying the answer, rephrasing the question, escalating to a human), and business outcomes (resolution rate).
Sample production traces for regular judge and human review, to catch drift the golden set doesn't cover.
A/B test larger changes; see A/B testing.

Where it’s used

RAG assistants

Retrieval recall and faithfulness tracked separately, with citation checks.

Agents

Sandboxed task-success suites with repeated trials and trajectory checks gate every release.

Model upgrades

The same golden suite decides whether a new model version replaces the current one.

In an interview

Define the eval plan early: dataset, graders per dimension, release gate.
Prefer code checks; use calibrated LLM judges for meaning; humans for calibration.
Evaluate RAG retrieval and generation separately.
Feed production failures back into the golden set.

Frequently asked questions

How do you evaluate an LLM application?

Build a golden dataset of realistic cases, grade outputs with code checks where possible, LLM judges calibrated against human labels for meaning, and human review for hard cases, gate releases on the results, and add production failures to the dataset.

Can you trust LLM-as-a-judge?

Only after calibration. Use specific rubrics, compare judge scores with human labels per dimension, control for position and length bias, and re-check when the judge model or rubric changes.

How do you evaluate a RAG system?

Measure retrieval and generation separately: retrieval recall and precision against known relevant passages, then faithfulness of the answer to the retrieved context, answer relevance, correctness and citation accuracy.