LLM application evals

What it is and why it exists

An LLM application eval is an executable quality specification for a feature whose output can vary. It supplies representative inputs, runs the actual application path, records outputs and traces, applies scoring rules, and compares the result with release thresholds. The unit under test is usually the application, not the base model alone.

That distinction matters. A retrieval assistant’s answer depends on document selection, prompt assembly, tool results, model settings, output parsing, and post-processing. A model benchmark cannot tell you whether your retriever chose an obsolete policy or your citation renderer attached the wrong source.

Ordinary unit tests still own deterministic contracts: schema parsing, permission checks, arithmetic, routing, and pure transformations. Evals cover behavior with several acceptable answers or qualities that need judgment, such as groundedness, task completion, tone, or refusal. Use both, and leave assertions in charge of facts that code can decide.

You meet evals when changing a prompt, model snapshot, retrieval index, tool definition, memory policy, or guardrail. They also turn production failures into durable cases. Without them, a polished demo can stand in for evidence even though it samples one prompt and one lucky generation.

The working artifact is a versioned corpus of cases and labels. Each case should say why it exists, which user or risk slice it represents, what inputs are fixed, what may vary, and how failure is decided. Store enough provenance to reproduce the application configuration even when the exact generated text cannot be reproduced.

The result is a decision system with row-level observations, aggregation rules, uncertainty, and explicit release policy. A useful result identifies which behaviors changed, by how much, for which users, under what configuration, and whether that change is allowed.

The contract of one case

A practical case has four layers. The input layer contains the user request and any controlled context. The execution layer identifies the application version, model configuration, tools, and trial count. The scoring layer defines checks and rubric labels. The decision layer says whether this case is informational, weighted, or a hard gate.

Keep expected behavior narrower than an expected paragraph. For a support answer, the contract might require the correct 30-day window, forbid invented order status, require a citation to the supplied policy, and cap the response length. Many phrasings can satisfy those observations.

Some cases need a reference answer; others do not. Exact extraction and classification often have stable references. Open-ended rewriting may be better judged by a rubric with anchored examples, while safety and access-control behavior usually needs deterministic invariants or human review.

The suite is a sampled product contract

No finite suite represents every production request. Treat case selection as sampling from important product behavior rather than collecting prompts that happen to be handy. Include common traffic, costly failures, boundary inputs, multilingual requests, long contexts, adversarial content, and previously observed incidents.

Name slices before seeing candidate results. Useful slices follow mechanisms or product obligations: retrieval depth, locale, customer tier, input length, tool path, or safety category. Post-hoc slices are valuable for diagnosis, but using only the most favorable one for a release claim is not a valid gate.

Cases need ownership and review. Product specialists define what a good answer does, engineers make execution reproducible, security staff own abuse cases, and annotators apply rubrics. When one person silently fills all four roles, hidden assumptions are likely to become labels.

How it works

An eval run should freeze everything you can control, sample what remains variable, and preserve evidence. At minimum, record the case-set revision, application commit, prompt or workflow revision, model identifier, inference parameters, tool fixtures, grader version, trial seed when supported, and run time.

A run proceeds in this order:

1. Select a locked evaluation set without looking at candidate outputs.
2. Run the baseline and candidate through the same inputs and controlled dependencies.
3. Capture final outputs plus relevant retrieval, tool, latency, token, and error data.
4. Apply deterministic checks before subjective graders.
5. Score surviving outputs with a rubric, human raters, or a calibrated judge.
6. Aggregate by named slice and compare every metric with its threshold.
7. Inspect changed rows before accepting or rejecting the candidate.

Pairing baseline and candidate on the same cases removes one source of noise: a different input mix. If generation is variable, run several independent trials for each case and configuration. Interleave requests when provider conditions or external tools may drift during a long run.

Keep raw observations distinct from derived metrics. One response may produce several observations: schema validity, citation precision, task completion, latency, and a human rating. Derive release metrics from those fields so you can change an aggregation rule without rerunning generation or losing the original evidence.

Make metric semantics explicit

The metric name must say what counts as one observation. case_pass_rate might mean that every criterion passed on one generation, while criterion_pass_rate averages checks within cases. Those numbers answer different questions even when both are displayed as percentages.

Multi-trial policies also need names. pass@k rewards getting at least one success from several attempts and fits products that may show or verify multiple candidates. pass^k asks whether every attempt succeeds and fits reliability claims. Neither describes a product that generates once unless the mapping to production behavior is justified.

For ranking or recommendation, decide whether the metric evaluates the chosen item, the entire ordered list, or the presence of any acceptable item. For agents, distinguish final-answer quality from trajectory properties such as unauthorized calls, excessive steps, and unrecovered tool errors. The final text can look right after an unsafe path.

Cost and latency are constraints alongside quality, not ingredients to mix into an unexplained score. Report their distributions and apply separate budgets. If a team chooses a utility formula, document its units and trade-off so a tiny quality change does not silently justify a large cost increase.

Build cases from behavior, not wording

Start with production task categories and failure records. For each category, write one ordinary case, boundaries that change the expected behavior, and counterexamples that look similar but should score differently. Deduplicate near copies so one popular template does not dominate the aggregate.

Synthetic cases help cover rare boundaries, but they can inherit the generator’s habits and miss messy user language. Keep their origin visible and compare them with sampled, consented production patterns. Remove personal data and secrets before they enter fixtures or grader prompts.

Hold back a set for release decisions. If you repeatedly tune prompts against every case, the suite becomes development data and its score becomes optimistic. A smaller visible development set plus a locked regression set gives fast feedback without pretending that repeated exposure has no effect.

Choose the cheapest trustworthy scorer

Use programmatic checks for facts that code can establish: JSON validity, required fields, exact identifiers, allowed citations, numeric tolerances, tool-call arguments, forbidden secrets, or execution success. These checks are fast, repeatable, and easy to debug.

Use a rubric when acceptable outputs cannot be enumerated. Define one criterion per score, state the evidence a rater may use, and include boundary examples. “Good response” is not a rubric; “states the policy window, distinguishes eligibility from approval, and makes no claim beyond the supplied policy” is.

Human review is appropriate for high-stakes meaning, disputed labels, and judge calibration. An LLM judge can scale a stable rubric after comparison with human labels, but it remains another model-based component. Version its prompt and model, test order effects, and keep an escalation path for low-confidence or release-blocking rows.

Aggregate without erasing failures

Calculate metrics at the case level before averaging. Report both the denominator and missing results; timeouts and grader errors must not vanish from the calculation. Decide beforehand whether infrastructure failures cause a rerun, count as application failures, or block the run as inconclusive.

Use weights only when they reflect a documented product priority or traffic model. Weights encode release policy and need justification. Publish unweighted and slice results alongside the weighted score so a large easy slice cannot conceal a small critical one.

Hard gates sit outside averages. Prompt-injection resistance, unauthorized tool use, personal-data disclosure, or a legally required statement may require zero known failures in the release set. Passing an overall threshold never compensates for breaking such an invariant.

Set regression thresholds before the run

A threshold translates scores into a release decision. It can require an absolute floor, limit a drop from baseline, demand improvement on a target slice, or combine all three. Write the direction, tolerance, minimum sample size, and treatment of uncertainty explicitly.

For example: “Grounded-answer pass rate must remain at least 92%, no named locale may regress by more than 3 percentage points, and every authorization case must pass.” This is testable. “Quality should be roughly as good” invites the decision to change after the scores arrive.

Small changes around a threshold are often noise. Retain trial-level results and compute an interval or use a paired test suited to the metric. If the data cannot distinguish an acceptable change from a forbidden regression, mark the run inconclusive and collect more observations rather than rounding toward release.

Preserve a decision record

The run summary should name the baseline, candidate, suite revision, grader revision, thresholds, and final status. Link every aggregate to case-level records. A reviewer should be able to recompute the decision without trusting prose generated by the harness.

Record warnings even when the run passes: low slice counts, judge disagreement, retried infrastructure errors, or changed external fixtures. These do not always block a release, but omitting them turns a conditional result into a stronger claim than the evidence supports.

Keep the decision separate from remediation. A failed gate says the candidate does not meet the current policy; it does not prove which prompt or component caused the change. Diagnose with traces and controlled experiments, then rerun the unchanged gate against the revised candidate.

Examples

The examples below score recorded candidate outputs, so they run without a provider key. In a production harness, the generation step writes the same kind of records with full configuration metadata. All shown output was produced with local Node 24.

Layering exact checks over flexible text

This first harness accepts varied prose but checks claims the product can decide mechanically. Each case names required and forbidden substrings; a real suite would also use structured parsers, citation IDs, and policy lookups.

The scorer is intentionally modest. Substring checks can detect the presence of a phrase, not whether the surrounding claim is correct. Treat each check as one signal and add a semantic or structured check when negation, attribution, or scope changes its meaning.

Notice that the cases test application behavior, not eloquence. The unknown-order answer refuses to invent status and gives a next step. A longer answer is not automatically better, and a reference string should not force every valid output into one wording.

Keeping hard gates outside the average

This release rule gives core invoice behavior more weight, yet keeps the injection case as a hard gate. The weighted score clears its floor, but the release still fails.

const results = [
  { id: "invoice-total", score: 1.0, weight: 4, hardGate: false },
  { id: "locale-date", score: 0.75, weight: 2, hardGate: false },
  { id: "prompt-injection", score: 0.0, weight: 1, hardGate: true },
  { id: "concise-tone", score: 1.0, weight: 1, hardGate: false },
];

const weightedPoints = results.reduce(
  (sum, result) => sum + result.score * result.weight,
  0,
);
const totalWeight = results.reduce((sum, result) => sum + result.weight, 0);
const weightedScore = weightedPoints / totalWeight;
const aggregatePass = weightedScore >= 0.8;
const hardGatesPass = results
  .filter((result) => result.hardGate)
  .every((result) => result.score === 1);

console.log(`weighted score: ${weightedScore.toFixed(3)}`);
console.log(`aggregate threshold: ${aggregatePass}`);
console.log(`hard gates: ${hardGatesPass}`);
console.log(`release: ${aggregatePass && hardGatesPass}`);

weighted score: 0.813
aggregate threshold: true
hard gates: false
release: false

The output exposes the two decisions separately. If the script printed only release: false, a reviewer could not tell whether broad quality or one invariant caused the block. Keep metric values, gate results, and failing case IDs in the run artifact.

In a mature suite, the weight and threshold live in reviewed configuration rather than inside an ad hoc script. Changing either one changes the product’s release policy and deserves the same review as a behavior change.

Catching a slice regression hidden by the total

These counts represent repeated case-trial observations. The candidate stays within the allowed three-point overall drop, but the long-context slice falls by fifteen points and breaches its five-point slice tolerance.

const slices = [
  { name: "routine", baseline: 36, candidate: 37, trials: 40 },
  { name: "long-context", baseline: 16, candidate: 13, trials: 20 },
  { name: "prompt-injection", baseline: 10, candidate: 10, trials: 10 },
];

const rate = (passed, trials) => passed / trials;
const total = (field) => slices.reduce((sum, slice) => sum + slice[field], 0);
const trials = total("trials");
const overallDelta = rate(total("candidate"), trials) - rate(total("baseline"), trials);

for (const slice of slices) {
  const delta = rate(slice.candidate, slice.trials) - rate(slice.baseline, slice.trials);
  const passed = delta >= -0.05;
  console.log(`${slice.name}: ${(delta * 100).toFixed(1)}pp, pass=${passed}`);
}

const overallPass = overallDelta >= -0.03;
const slicesPass = slices.every(
  (slice) => rate(slice.candidate, slice.trials) - rate(slice.baseline, slice.trials) >= -0.05,
);
console.log(`overall: ${(overallDelta * 100).toFixed(1)}pp, pass=${overallPass}`);
console.log(`release: ${overallPass && slicesPass}`);

routine: 2.5pp, pass=true
long-context: -15.0pp, pass=false
prompt-injection: 0.0pp, pass=true
overall: -2.9pp, pass=true
release: false

Percentage-point changes compare rates directly; they are not percent changes. The example’s sample is too small to support a confident production decision by itself. A real report should attach intervals and the underlying paired rows, then collect more trials when the interval crosses the permitted regression boundary.

The fixed slice threshold also prevents a tempting response to failure: redefining “long context” until the result passes. Slice definitions and minimum denominators belong in the suite revision used before generation.

Pitfalls

Tuning against a demo set

Fix: separate development examples from a locked release set, add cases from incidents and sampled task categories, and report results by declared slice. Rotate or expand the locked set under review without exposing its labels to the prompt-tuning loop.

Using one vague judge score

Fix: split the rubric into observable criteria with anchored score levels. Calibrate judge decisions against blinded human labels, record disagreement, and route critical or ambiguous rows to human review.

Letting averages compensate for forbidden behavior

Fix: express non-compensable behavior as a hard gate and keep it outside weighted aggregation. Print failed gate IDs, preserve their traces, and require explicit remediation or risk acceptance rather than hiding them in a decimal score.

Treating one generation as the case result

Fix: pair configurations on identical inputs, repeat trials at a count justified by decision risk, and retain trial-level observations. Use intervals or an appropriate paired analysis, and allow an inconclusive state when evidence is too weak.

Contaminating cases or grader context

Fix: separate generation inputs from grader-only fields, restrict access to locked labels, sanitize production-derived cases, and log exactly which fields each component receives. Rebuild compromised cases instead of continuing to compare against a leaked baseline.

Further reading

• OpenAI: Evaluation best practices
• Anthropic: Define success criteria and build evaluations
• Google Cloud: Gen AI evaluation service overview
• Microsoft Foundry: General purpose evaluators for generative AI