Building LLM Evaluation Pipelines

If You Remember Nothing Else

You don't need to build evaluation infrastructure from scratch. Mature tools cover most needs:

• Ragas for retrieval and generation metrics (faithfulness, context precision, answer relevance).
• Langfuse for production observability (trace every request, sample for evaluation).
• Weights & Biases or ClearML for experiment tracking integrated with model registry.
• DeepEval for pytest-style evaluation in CI.

Custom harnesses make sense only when off-the-shelf doesn't fit your specific needs. The biggest mistake we see is teams shipping models without any evaluation pipeline, then trying to debug quality issues with no telemetry.

The right pipeline runs evaluation on every model change in CI, samples production traffic for drift detection, and stores historical metrics so regressions are visible.

Recommendations by Situation

Worked Examples

Example 1: Standard RAG eval pipeline (Ragas + Langfuse)

A B2B knowledge product uses RAG. Team wants to systematically measure quality.

The right approach:

• Ragas for offline evaluation: faithfulness, context precision, context recall, answer relevance. Run on every model or RAG pipeline change. Test set: 200 curated queries.
• Langfuse for production: every request traced, 1% sampled for evaluation. Same metrics computed on production samples.
• CI integration: PRs that drop any metric by more than 2 points get flagged for review.

Total setup: 2 weeks. Engineering investment: ~$5K. Ongoing: $200 per month for Langfuse Cloud + LLM judge costs.

What worked: standard tools, standard metrics. The team didn't reinvent. Quality was measurable on day one.

What they nearly got wrong: building custom evaluation. The first proposal was a 6-week custom harness. Ragas + Langfuse delivered the same outcome in 2 weeks at near-zero ongoing cost.

What to remember: standard tools cover 90% of evaluation needs. Build custom only when standard doesn't fit.

Example 2: Multi-model A/B evaluation (Promptfoo)

A team wants to compare GPT-4o, Claude 3.5 Sonnet, and a fine-tuned Llama 70B on the same task.

The right approach: Promptfoo. Define test cases (50 representative queries) and quality criteria (faithfulness, completeness, helpfulness). Run all three models against the same suite. Get a side-by-side comparison report.

Setup time: 2 days. Result: fine-tuned Llama 70B beat GPT-4o on the specific task by 4 points; Claude 3.5 was close to Llama. Cost analysis showed Llama was 50x cheaper at sustained volume.

What worked: comparative evaluation as a first-class tool. Promptfoo is purpose-built for "compare these models on these tests."

What they nearly got wrong: building custom comparison logic. The team had been running each model separately and comparing manually. Three days of work; Promptfoo did it in 30 minutes.

What to remember: when comparing across models or prompts, use a tool designed for it. Manual comparison wastes time and misses subtleties.

Example 3: Production drift detection (custom dashboards)

A platform has 5 different AI features in production. Volume: 50M tokens per day. Need to detect drift before users complain.

The right approach: combine multiple tools.

• Langfuse traces every production request.
• 0.5% of requests sampled for LLM-as-judge grading on 4 quality dimensions.
• Daily aggregates pushed to Grafana dashboards (per-feature, per-tenant breakdowns).
• Alerts trigger when a metric drops below thresholds.

Setup time: 4 weeks. Ongoing cost: ~$2K per month for sampling + judge tokens + dashboard infrastructure.

What worked: combining tracing tool (Langfuse) with grading tool (LLM-as-judge) and dashboard tool (Grafana). Each layer has a specific job.

What they nearly got wrong: sampling at 5% (10x more than needed). The cost of grading would have been 10x higher with no additional signal. 0.5% was statistically sufficient.

What to remember: production sampling is about catching drift, not exhaustive measurement. A small sample is enough; spending more on judging buys diminishing returns.

Anti-Patterns to Watch For

"We'll build a custom evaluation system"

What it looks like: ambitious in-house evaluation infrastructure plans.

Why it's wrong: 90% of needs are covered by Ragas, Langfuse, DeepEval, Promptfoo, or similar. Custom is weeks of work for outcomes the tools deliver in days.

How to redirect: prototype with standard tools first. Build custom only for genuine gaps.

"Evaluation runs only when we ship"

What it looks like: evaluation as a release gate, not part of development.

Why it's wrong: by ship time, regressions are expensive to fix. Catching them mid-PR is cheap.

How to redirect: integrate evaluation into CI. Every PR runs the eval suite. Quality regressions block merge.

"LLM-as-judge is enough"

What it looks like: skipping human validation entirely.

Why it's wrong: LLM judges have systematic biases. Without periodic human calibration, the judge becomes unreliable over time.

How to redirect: weekly or monthly, sample 20 to 30 LLM-judged cases for human review. Compare. Recalibrate the judge if drift appears.

"Production sampling is too expensive"

What it looks like: avoiding production grading due to cost.

Why it's wrong: 0.5 to 1% sampling is statistically sufficient. The grading cost is small relative to base inference cost.

How to redirect: do the math. At 50M tokens per day, 1% sampling is 500K tokens per day to grade. At GPT-4 prices, that's $25 per day. Cheap relative to the value of catching drift.

"Evaluation is just for the launch"

What it looks like: building eval infrastructure for one launch and abandoning it.

Why it's wrong: continuous evaluation catches drift, regressions, and production-specific issues. The infrastructure pays back recurring.

How to redirect: institutionalize evaluation as ongoing process. Weekly review of metrics. Quarterly review of test set coverage.

When NOT to Invest in Evaluation Tooling

Specific cases where lightweight is sufficient:

• Truly experimental prototype with no commitment to production.
• Internal-only tool, low criticality, low volume.
• Pre-product-market-fit; evaluation infrastructure for a feature you might kill is wasted.
• Single-developer side project.

In these cases, manual spot-checks of 10 to 20 cases is enough.

What to Ask Your Engineering Team

1. What evaluation tool are we using? Off-the-shelf or custom?
2. Where does evaluation run? Local dev only, CI, production sampling, all three?
3. What's the test set, and how is it maintained?
4. How is the LLM judge calibrated against humans?
5. What metrics fail what threshold trigger what action?
6. Is production traffic being graded continuously?
7. Can we see metrics over time, by feature, by tenant?

Cost & Timeline Quick Reference

Realistic ranges for evaluation infrastructure:

Most teams should start with Ragas + Langfuse and add specialized tools as needs emerge.

The Bottom Line

Use standard tools. Ragas for metrics, Langfuse for production observability, DeepEval for CI, Promptfoo for comparisons. Build custom only for genuine gaps.

Evaluation pipelines run on every model change, sample production traffic continuously, and surface metrics over time. Without this infrastructure, the team can't tell if changes helped, hurt, or did nothing.

The cost is moderate; the value is permanent. Teams that invest in evaluation tooling ship more reliable AI; teams that skip it debug quality issues blindly.