Evaluating Model Quality Beyond Accuracy

If You Remember Nothing Else

"Accuracy" as a single number is the wrong way to measure production model quality. Real production quality is multi-dimensional: faithfulness (the model said only what the source supports), helpfulness (the answer addressed the actual question), calibration (the model knows when it doesn't know), latency, cost per useful response, and safety.

Track these per-category, not just as averages. A 90% average can hide 50% on a critical query class. The averages look fine while specific user segments experience broken product features.

The evaluation harness should be built before fine-tuning or any optimization work starts. Without it, the team can't tell if changes helped, hurt, or did nothing.

Recommendations by Situation

Worked Examples

Example 1: RAG product evaluation suite (multi-dimensional)

A B2B knowledge product uses RAG over enterprise documents. Quality matters because users rely on it for compliance decisions.

The right approach: 4-dimensional evaluation harness, run on every model change.

• Faithfulness: does the response only assert claims supported by the retrieved context? (LLM-as-judge with ground truth.)
• Citation accuracy: are the cited sources actually the ones containing the answer?
• Refusal correctness: does the model say "I don't have enough information" when context is insufficient?
• Latency: end-to-end response time (target sub-3 seconds).

Test set: 200 manually curated queries across 12 question categories.

What worked: catching regressions that simple accuracy would have missed. A model upgrade improved overall accuracy from 87% to 89% but dropped faithfulness from 91% to 78% (more hallucination). Without the dimensional harness, the team would have shipped the regression.

What they nearly got wrong: optimizing for accuracy alone. The ChatGPT-style "always give an answer" failure mode has its own metrics; without them, you optimize the wrong thing.

What to remember: define quality as multiple specific dimensions before evaluation starts. "Accuracy" alone is misleading.

Example 2: Per-category breakdown reveals hidden failure (classification system)

A support classifier routes tickets into 14 categories. Overall accuracy: 91%. The team was about to ship.

The right approach: per-category breakdown. Result: most categories at 95%+, but one category (refund disputes) at 47%. The model was systematically misrouting refund disputes to a wrong category.

Root cause: training data was unbalanced (refund disputes were under-represented).

What worked: discovering this before shipping. The PM had insisted on per-category metrics; the engineering team had only been looking at the average.

What they nearly got wrong: shipping based on average. 91% sounds great. Refund disputes routing wrong would have caused 100+ angry customers per week.

What to remember: never accept averages as quality measure. Always demand per-category breakdowns. The minority class is often the most critical class.

Example 3: Production sampling detects drift (live monitoring)

A content moderation model has been in production for 6 months. Average quality at deployment was 94%. Six months later, no one knows the current quality.

The right approach: production sampling. Sample 1% of traffic, manually grade by trained reviewers, track metrics over time.

Result: model quality had drifted from 94% to 87% over 6 months as user content patterns evolved. The drift was gradual (2% per month) and invisible without explicit measurement.

What worked: institutionalizing production quality measurement. The team set up a permanent process: 200 random samples per week, graded by 2 reviewers, tracked in a dashboard.

What they nearly got wrong: assuming "deployed once, working forever." Models drift; production data drifts; without measurement, quality silently degrades.

What to remember: ongoing production sampling is not optional. Build it into the operational rhythm.

Anti-Patterns to Watch For

"Accuracy is good enough"

What it looks like: single-metric quality reporting.

Why it's wrong: production quality is multi-dimensional. Accuracy hides hallucination, latency, calibration failures, and category-specific issues.

How to redirect: insist on at least 3 quality dimensions per AI feature. Define them before evaluation starts; don't add them after results come in.

"We'll evaluate after we ship"

What it looks like: deferring evaluation until production.

Why it's wrong: shipping without evaluation infrastructure means you can't measure changes. Quality regressions go undetected.

How to redirect: build the evaluation harness before fine-tuning or optimization starts. Use it to validate every change.

"The benchmarks say we're at 92%"

What it looks like: relying on published benchmark scores.

Why it's wrong: published benchmarks rarely match your specific task. A model that scores 92% on MMLU may score 60% on your support classification task.

How to redirect: build a task-specific benchmark from your real or expected production traffic. Published benchmarks are useful for screening; task-specific is what matters.

"We use LLM-as-judge so it's automated"

What it looks like: replacing human evaluation entirely with LLM-as-judge.

Why it's wrong: LLM judges have systematic biases (favoring verbose responses, position bias, calibration drift). Without occasional human validation, the judge can be wrong while looking reliable.

How to redirect: use LLM-as-judge for scaled grading, but periodically (weekly or monthly) validate the judge's grades against human review on a sample. Recalibrate when needed.

"We track quality only on the test set"

What it looks like: evaluation that runs against a fixed benchmark set, never against production.

Why it's wrong: production data drifts. The test set captures the past; current production may have different patterns.

How to redirect: combine offline benchmarks (locked test set) with production sampling (1 to 5% of live traffic graded continuously). Both signals are necessary.

When NOT to Build a Full Evaluation Harness

Specific cases where the answer is lighter-weight evaluation:

• The product is an experimental prototype. Heavy evaluation infrastructure for an MVP is over-engineering.
• The use case is non-critical (internal tool, low-volume, no user impact).
• The team genuinely cannot allocate the engineering time. A poorly-built evaluation is worse than none; build it well or wait.
• The model is a frontier API and you have no fine-tuning capacity. Provider-level changes are out of your control; focus on prompt engineering.

In these cases, lightweight LLM-as-judge with a few dozen examples is sufficient. Build the full harness when scale or criticality justifies it.

What to Ask Your Engineering Team

1. What are the 3 to 5 quality dimensions we measure for this AI feature?
2. What's the test set, and how was it constructed? Manual curation, real production sampling, both?
3. How is the test set protected from contamination?
4. What's the per-category breakdown of quality? Not just averages.
5. How do we detect production drift? Specific monitoring plan.
6. What's the LLM-as-judge calibration story? Periodic human validation?
7. What's the regression process when a model change causes quality drop?

Cost & Timeline Quick Reference

Evaluation harness investment:

The harness pays back via faster iteration and prevented quality regressions. Without it, every model change is a guess.

The Bottom Line

Evaluate model quality across multiple dimensions, never just accuracy. Always per-category, never just averages. Combine offline benchmarks with production sampling.

Build the evaluation harness before fine-tuning or optimization starts. Without it, the team is flying blind. With it, every change is measured and the priority order of improvements becomes clear.

The teams that ship reliable AI aren't the ones with the cleverest models. They're the ones with the most rigorous evaluation.