Evaluation & AI Quality Engineering

1. Quality is a decision system, not a score

An evaluation system exists to make decisions: continue iterating, merge a change, canary it, expand rollout, roll back, investigate a slice, or escalate a risk. Start by writing the decision and the consequence of a false pass or false fail. Only then choose metrics and thresholds. A single “quality score” cannot represent correctness, evidence use, safety, latency, cost, and user value without hiding important trade-offs.

Build a quality tree

For an enterprise policy assistant, the top outcome might be “authorized users resolve policy questions accurately and quickly.” Decompose it into: relevant current evidence retrieved; answer claims supported; citations resolvable; uncertainty handled; unauthorized information never disclosed; correct escalation on ambiguous or high-risk questions; and acceptable latency/cost. Attach at least one measure and one failure example to each leaf.

2. Engineer the dataset before the evaluator

A golden set is a versioned collection of inputs, expected properties, metadata, and judgments. “Golden” means reviewed, traceable, and stable enough for comparison—not perfect or frozen. OpenAI recommends task-specific tests that reflect real distributions and continued human calibration; LangSmith and Langfuse both support offline-to-production feedback loops.

Four complementary sources

1. Curated core: expert-written canonical and boundary cases.
2. Production traces: privacy-reviewed samples of common, failed, costly, uncertain, and novel interactions.
3. Adversarial cases: injection, leakage, malformed input, unavailable tools, contradictory sources, and no-answer examples.
4. Synthetic expansion: reviewed, provenance-labeled paraphrases or rare combinations; supplements, not proof of representativeness.

Each record should include stable ID, input, reference evidence or expected behavior, rubric, risk, source/provenance, created/reviewed dates, language, tenant/data class, intent, difficulty, and applicable evaluators. Preserve an immutable raw trace reference separately when allowed. Redact or synthesize sensitive values before placing cases in developer-visible stores.

Split by how data can leak

Keep development examples for iteration, validation for thresholds, and a sequestered test set for final claims. Group by source document, user/thread, template, or time so near-duplicates do not cross splits. Repeated optimization makes validation data de facto training data; preserve a final untouched set.

Version everything that changes meaning

Version dataset, judgments, rubrics, corpus/parser/retrieval, prompt/model/tools, evaluators, sampling, and environment. Store hashes and a changelog. A score without dataset and evaluator versions is not reproducible evidence.

3. Match metrics to the failure boundary

Component scores localize defects; end-to-end scores reveal interactions. Retrieval recall can rise while excess context harms answers; correct tool selection can still carry invalid arguments.

Retrieval and RAG

• Retrieval: precision@k, recall@k, MRR, nDCG, filtered slices, and evidence coverage after packing.
• Grounding: claim-level support by retrieved evidence, citation correctness, citation completeness, and contradiction.
• Answer: task correctness/completeness, relevance, instruction adherence, appropriate uncertainty, and abstention.

Faithfulness and correctness differ: an answer can repeat stale evidence faithfully or be correct yet unsupported. Evaluate both. Ragas exposes RAG and agent metrics, but inspect definitions, prompts, model dependence, and intermediate outputs before using any as a gate.

Agents and tool use

• Goal/task completion and correct terminal state.
• Tool-selection precision/recall and unnecessary-call rate.
• Argument schema and semantic validity; authorization/policy result.
• Trajectory efficiency: calls, retries, replans, repeated steps, tokens, and wall time.
• Side-effect correctness, duplicate effects, recovery, and escalation.

Deterministic output checks first

Use exact match for known classifications, JSON Schema for structure, parsers/compilers for code or queries, executable tests for calculations, database comparisons for extraction, and policy engines for allowed actions. A deterministic checker is cheaper, faster, repeatable, and easier to debug than a model judge when the property is mechanically decidable.

4. Calibrate human and model judgment

Human review is nuanced but slow and variable. Model judges scale but inherit prompt sensitivity, bias, position/verbosity preferences, and domain blind spots. Prefer deterministic checks, use humans for ground truth/high-risk ambiguity, and calibrate model judges for broader coverage.

Write an operational rubric

Replace “good answer, 1–5” with observable anchors. For faithfulness, label material claims supported, contradicted, or absent from evidence. Define required and harmful task behavior, give boundary examples, and permit “insufficient information.”

Measure annotator behavior

Train reviewers on shared cases, blind variant identity, randomize order, and double-label a stratified subset. Track agreement by criterion/slice and adjudicate. Low agreement can expose training, rubric, evidence, or unsettled-requirement problems.

Calibrate an LLM judge

1. Build a held-out, human-labeled calibration set with hard and boundary cases.
2. Give the judge only the information the rubric requires; prevent candidate metadata from revealing the variant.
3. Prefer categorical or pairwise decisions with explicit anchors when absolute numeric scoring is unstable.
4. Test candidate-order reversal, verbosity, self-preference/provider, reference leakage, and prompt-injection inside the content being graded.
5. Measure agreement, confusion matrix, false-pass rate on high-risk cases, and stability across repeated runs.
6. Version judge model, prompt, temperature/settings, rubric, and calibration result; recalibrate after change.

Do not let the judge’s explanation substitute for correctness. Store reasoning as debugging material, not proof. If a judge is weak on a critical slice, route that slice to a deterministic check or human review. For release gates, prioritize the false-pass cost: an evaluator that misses unsafe behavior is worse than one that occasionally sends a safe run for review.

5. Turn failures into an error taxonomy

Aggregate scores show population movement; error analysis chooses the repair. Give each failure a primary stage, symptom, likely cause, consequence, and owner. Secondary tags can capture interactions.

Slice before celebrating

Choose slices from risk and plausible causes: intent, language, policy regime, tenant/data class, document type, exact identifier, multi-hop, no-answer, tool, route, cohort, and age. Report counts and uncertainty; define critical slices before the experiment.

Inspect paired deltas

List paired improvements and regressions. Equal means can hide replacing harmless style errors with one security failure. Inspect largest negative deltas and every invariant violation; track severity and consequence.

6. Build release gates that tolerate variability, not regressions

CI compares immutable baseline and candidate configurations on a pinned dataset/evaluator suite. Record environment and inspectable responses when policy permits. Invalidate caches for every changed prompt, model, retrieval, or tool dimension.

def release_decision(base, candidate):
    hard_fail = any(candidate[name] != 1.0 for name in (
        "tenant_isolation", "schema_valid", "no_duplicate_effect"
    ))
    quality_drop = candidate["task_success"] < base["task_success"] - 0.02
    slow = candidate["p95_ms"] > 1.10 * base["p95_ms"]
    costly = candidate["cost_per_success"] > 1.15 * base["cost_per_success"]
    return "block" if hard_fail or quality_drop or slow or costly else "canary"

# Thresholds above are illustrative; derive real gates from product risk.

Hard invariants require every applicable case to pass. Comparative metrics need both absolute floors and allowable deltas. Critical slices need their own gates. Cost and latency are guardrails. Treat missing evaluator output as a failure or explicit “inconclusive,” never as a pass. Keep a small smoke suite on every change and a larger suite on scheduled runs or release candidates, while ensuring high-risk cases remain in the fast gate.

Account for stochastic and sampling uncertainty

Use paired comparisons and report intervals via paired bootstrap or another appropriate method; inspect discordant binary outcomes. Repeat a stratified subset to estimate variance. Never average away safety failures, and weigh practical—not only statistical—significance.

Move online carefully

Shadow, then canary or A/B test using a randomization unit that avoids contamination. Predefine outcome, guardrails, exposure, stop rules, and analysis; monitor assignment and segment regressions. Never expose an unsafe variant for statistical power.

7. Close the offline–online quality loop

Offline datasets provide controlled repeatability; production provides distribution reality. Instrument traces with application, prompt, model, retrieval, tool, evaluator, and release versions plus safe outcome metadata. Sample common traffic randomly for prevalence, and oversample rare/high-risk signals for discovery. Keep weighting clear: a risk-enriched review queue cannot estimate population quality without correcting its sampling design.

Production signals are evidence, not ground truth

Track feedback, completion, abandonment, reformulation, escalation, citations, corrections, latency, cost, and denials. Each is confounded: clicks reflect position and silence may mean abandonment. Calibrate proxies with reviewed traces.

Choose tools by data model and exit path

Keep manifests and core evaluators portable. Export scores, case/trace references, and versions. OpenTelemetry semantics evolve and content can be sensitive; pin conventions and avoid message bodies by default.

8. Make safety and privacy executable

Build adversarial cases from actual inputs: user text, retrieval, tools, files, connectors, memory, and tenant data. Use the NIST Generative AI Profile to structure governance, then translate it into system-specific controls and tests.

Test the full threat path

• Prompt injection: direct user instructions and indirect instructions embedded in retrieved pages, documents, tool output, or memory.
• Sensitive information: secrets, PII, hidden prompts, credentials, and private records requested directly or inferred through side channels.
• Tenant isolation: IDs from another tenant, mixed-index candidates, cached responses, trace views, and shared memory.
• Unsafe tool use: unauthorized tool, excessive scope, altered action after approval, malicious URL/arguments, duplicate effect, and ambiguous timeout.
• Policy behavior: refusal consistency, over-refusal on benign requests, safe alternatives, and correct human escalation.
• Robustness: malformed encoding, extreme length, empty/contradictory evidence, unavailable dependencies, and partial streaming.

Measure attack success rate, sensitive-data disclosure, unauthorized-action rate, false refusal, escalation precision/recall, time to detection, and recovery behavior by attack surface. Hard security boundaries should be enforced deterministically and must pass every applicable test. Model-based red teaming can expand cases, but manually validate coverage and preserve sequestered attacks to reduce test gaming.

Protect the evaluation system itself

Datasets and traces contain revealing failures. Apply minimization, access control, encryption, retention/deletion, tenant partitioning, and audit. Treat candidate text as untrusted judge input; sandbox code evaluators with minimal permissions. Send production content externally only under an approved data contract.

Syllabus checkpoint: metric language and evaluation tooling

Faithfulness, groundedness, and relevance

Teams use these labels inconsistently, so define the rubric before quoting a score. A practical convention is: faithfulness asks whether answer claims are supported by supplied evidence; groundedness asks whether the answer is anchored to the authorized source context rather than unsupported model knowledge; relevance asks whether the response addresses the user’s actual request. Add completeness and citation correctness separately—a faithful answer can still omit the decisive exception.

Deterministic checks, probabilistic scoring, and human annotation

Use deterministic checks for schemas, required fields, citations that resolve, forbidden strings, permissions, exact calculations, and known invariants. Use probabilistic scoring for semantics that tolerate legitimate variation, but calibrate thresholds and uncertainty. Human annotation needs an operational rubric, examples near decision boundaries, blinded ordering where possible, adjudication, agreement measurement, and recorded annotator context. Sampling must represent important slices rather than only easy or recently failed cases.

Choose a platform by the evaluation loop, not the dashboard

LangSmith and Langfuse can manage traces, datasets, experiments, and feedback with different hosting and ecosystem trade-offs. RAGAS and DeepEval provide useful evaluation building blocks; custom Python harnesses remain valuable for exact product contracts. Arize Phoenix and commercial Arize capabilities add tracing/evaluation and observability workflows. OpenTelemetry supplies vendor-neutral trace context and export. Regardless of tool, require dataset, prompt, model, judge, code, and environment versions plus links from aggregate regressions to inspectable traces.

Interview playbook

Use QUALITY to answer an evaluation-system design prompt:

1. Q — Question and consequence: Which release/product decision, and what does a false pass cost?
2. U — User distribution: traffic, important slices, risks, and no-answer/edge behavior.
3. A — Artifacts and annotations: dataset sources, provenance, rubric, splits, versions, and privacy.
4. L — Layers of measures: deterministic, component, end-to-end, human, judge, safety, and operations.
5. I — Inspect errors: paired deltas, taxonomy, severity, slices, uncertainty, and evaluator failures.
6. T — Threshold and trial: hard invariants, regression gates, shadow/canary/A-B design, rollback.
7. Y — Yield feedback: production sampling, adjudication, new cases, ownership, and change cadence.

Common traps

• Choosing metrics before defining the product decision and failure cost.
• Using one aggregate judge score with no rubric, calibration, or slice analysis.
• Calling synthetic questions representative without validation against production.
• Tuning prompt, threshold, and evaluator on the same test set and reporting it as generalization.
• Allowing quality gains to compensate mathematically for security or privacy violations.
• Comparing unpaired runs with changed corpus, parser, environment, and model all at once.
• Treating user feedback, clicks, or judge explanations as uncontested ground truth.
• Building dashboards without an owner, alert/action threshold, or rollback path.

Question bank

These questions test whether evaluation evidence can support a production release decision.

Proof artifact: a versioned RAG release gate

Build a local, vendor-neutral evaluation harness for the public-data RAG system from chapter 2. It should compare baseline and candidate, publish case-level diffs, and return a failing process status when a release gate is violated. All thresholds and results are portfolio examples, not claims about Purnendu’s work history.

Steps

1. Create a JSONL or table-based dataset with stable IDs, input, reference evidence, expected properties, slice tags, risk, provenance, and split. Hash the manifest.
2. Run a frozen baseline and one candidate on the same validation cases. Capture retrieval IDs, packed context, response, citations, tool/trace events, latency, tokens, cost estimate, and all versions.
3. Implement deterministic schema, citation-resolution, tenant-isolation, and no-duplicate-effect checks; retrieval metrics; and a human-calibrated claim-support judge.
4. Generate overall and slice tables, paired deltas, error taxonomy, invariant failures, and Pareto plots. Persist evaluator failures separately.
5. Encode release rules: all hard invariants pass, minimum quality floor, maximum allowable regression overall and on critical slices, and latency/cost budgets.
6. Run the small critical suite in CI. Schedule the full suite and require the report for canary promotion. Document owner and rollback trigger.

Metrics

Capture recall@20, nDCG@10, packed evidence coverage, claim support, task success, citation correctness, abstention, safety invariants, evaluator agreement on a calibration subset, p50/p95/p99 latency, tokens, cost per successful task, and failures by taxonomy/slice. Include sample counts and uncertainty.

Deliberate failure injection

• Insert one cross-tenant evidence item and prove the invariant blocks the release even if average relevance rises.
• Alter the judge prompt to favor verbose answers and demonstrate calibration/order tests detect drift.
• Remove citations after generation and verify deterministic resolution/completeness checks fail.
• Slow the reranker and confirm the latency guardrail catches the p95 regression.
• Leak near-duplicate source questions across splits, observe the inflated score, then repair grouping and document the change.

What to present

Show the quality tree, dataset card/manifest hash, evaluator calibration matrix, baseline-versus-candidate slice report, one blocked CI run, two error traces, and the exact release decision. The strongest demonstration is a candidate with a better aggregate score that the gate correctly refuses because a critical invariant or slice regressed.

Chapter review

Evaluation engineering turns variable model behavior into bounded release decisions. It begins with product consequences and representative data, uses the cheapest valid evaluator at each boundary, calibrates human and model judgment, inspects errors and slices, and connects offline evidence to guarded production rollout. The evaluation system itself is versioned, tested, monitored, and protected.

Glossary

Golden set

A reviewed, versioned dataset with inputs, provenance, expected properties/evidence, metadata, and judgments for repeatable comparison.

Guardrail metric

A limit that an optimization may not violate, such as p95 latency or cost per successful task.

Hard invariant

A property that must always hold, such as tenant isolation; it is not averaged with softer quality metrics.

LLM-as-judge

A model-driven evaluator that applies a rubric to an output or comparison and must be calibrated like any other model component.

Paired evaluation

Comparison of variants on the same cases, enabling per-case deltas and more efficient uncertainty analysis.

Slice

A meaningful subset defined by risk, user group, intent, data, or causal mechanism.

Faithfulness

Whether response claims are supported by supplied evidence; distinct from real-world correctness.

Evaluation leakage

When test information influences development or related examples cross splits, inflating apparent generalization.

Mastery checklist

• I can draw a product-specific quality tree and label objectives, guardrails, and invariants.
• I can design dataset sources, provenance, grouped splits, slices, and a version manifest.
• I can distinguish retrieval, packing, grounding, answer, agent, and operational metrics.
• I choose deterministic evaluators whenever the property is mechanically decidable.
• I can calibrate a model judge against blinded human labels and test its biases.
• I can turn case failures into an actionable taxonomy and paired slice report.
• I can specify CI gates, uncertainty handling, shadow/canary, and rollback.
• I can design injection, leakage, tenant, tool, refusal, and escalation tests.
• I can compare evaluation platforms while preserving portability and data governance.