Monitoring LLM Systems in Production

If You Remember Nothing Else

Production LLM monitoring isn't optional. The cost surprises, quality regressions, and reliability incidents that hit unmonitored AI systems are real and recurring.

Five metric layers matter:

• Cost (per request, per feature, per tenant). Drives capacity planning and unit economics.
• Latency (p50, p95, p99 by endpoint). Drives user experience.
• Quality (sampled grading, drift detection). Drives trust.
• Reliability (error rates, fallback chain engagement, provider availability). Drives uptime.
• Safety (refusal rates, policy violations, prompt injection attempts). Drives risk.

If your team can't show you the dashboard for any of these on demand, that's the gap to fix first. Cost surprises in particular are caused by missing instrumentation, not bad architecture.

Recommendations by Situation

Worked Examples

Example 1: Cost monitoring catches budget surprise (high-volume SaaS)

A SaaS product processes 100M tokens per day across multiple AI features. Monthly bill: $35K. The team had no per-feature cost breakdown.

The right approach: Langfuse traces every request with feature tag. Daily aggregate dashboards show cost-per-feature, cost-per-tenant, cost-per-query type. Budget alerts fire when daily spend exceeds projection by 20%.

What worked: discovering that one feature ("AI summarization for newsletter") was consuming 40% of cost while delivering low engagement. Team made a product decision to deprecate the feature; saved $14K per month.

What they nearly got wrong: spending another $14K per month on a feature with low value. Without per-feature cost attribution, no one knew how to make the trade-off.

What to remember: per-feature cost attribution is the single highest-value monitoring investment. Without it, cost optimization is guessing.

Example 2: Quality drift detection (production AI for 6 months)

A content moderation model has been in production for 6 months. Average quality at deployment: 94%. Six months later, no one knew the current quality.

The right approach: production sampling. 0.5% of traffic graded daily by LLM-as-judge with weekly human calibration. Time-series dashboard tracks quality trends per content category.

Result: model quality had drifted from 94% to 87% over 6 months as user content patterns evolved. Drift was gradual (1 to 2 points per month); easily missed without explicit measurement.

What worked: institutionalizing production quality measurement. The drift triggered re-training; quality recovered to 93% after retraining.

What they nearly got wrong: assuming "deployed once, working forever." Models drift; production data drifts; without measurement, quality silently degrades.

What to remember: continuous production sampling is required for AI systems. Quality degradation is gradual and invisible without instrumentation.

Example 3: Latency tail catches UX-breaking events (real-time voice)

A voice assistant has p50 latency of 200ms (good) but the team didn't track p99. Users were complaining about occasional "long pauses."

The right approach: detailed latency monitoring. P50, p95, p99 tracked per endpoint. Tail latency events (over 2 seconds) logged with full traces.

Result: p99 latency was 4 seconds. About 3% of requests hit a slow path due to a misconfigured caching layer. The "occasional pauses" were these 3% events. Users perceive them disproportionately because they break conversational flow.

What worked: tail latency monitoring caught what averages hid. Fix took two days; users reported substantial UX improvement.

What they nearly got wrong: relying on average latency. Average looked fine; UX was actually broken for a small but meaningful slice of requests.

What to remember: in user-facing AI, p99 matters more than p50. Track tail latency explicitly; investigate every regression.

Anti-Patterns to Watch For

"We have logs, we're good"

What it looks like: monitoring infrastructure consisting of raw logs, no aggregation or alerting.

Why it's wrong: logs are debugging tools, not monitoring tools. Without aggregation, dashboards, and alerting, the team won't notice problems until users complain.

How to redirect: invest in actual monitoring (Langfuse, Datadog, custom Grafana dashboards). Logs are a layer below; monitoring is what catches issues.

"We'll monitor when it becomes a problem"

What it looks like: deferring monitoring until something breaks.

Why it's wrong: by the time it breaks, you don't have the data to understand what broke. Monitoring is most valuable for the failure that hasn't happened yet.

How to redirect: ship monitoring with the AI feature, not after. Cost telemetry, latency monitoring, and basic quality sampling are all setup-day tasks.

"Cost is just whatever it ends up being"

What it looks like: no cost telemetry; bill is whatever the provider invoices.

Why it's wrong: at any meaningful scale, cost optimization opportunities are huge. Without cost monitoring, you can't find them.

How to redirect: tag every AI request with feature, tenant, query type. Aggregate. Surface in dashboards. The first time you do this, expect to find 30 to 50% of cost concentrated in 2 to 3 surprises.

"We monitor averages, that's enough"

What it looks like: dashboards showing only mean values.

Why it's wrong: averages hide tail events. P99 latency events break UX. Worst-case quality on minority categories drives user complaints.

How to redirect: track p50, p95, p99 for latency. Track per-category breakdowns for quality. Averages are the start, not the end.

"Alerts are noisy, we ignore them"

What it looks like: alerts that fire too often, leading to alert fatigue.

Why it's wrong: noisy alerts mean real issues get missed. The team learns to tune out the noise, including the signal.

How to redirect: tighten thresholds until alerts are actionable. Better to alert less and respond more than alert constantly and respond never.

When NOT to Build Heavy Monitoring

Specific cases where lightweight is sufficient:

• Single-developer prototype with no users.
• Internal-only tool with low criticality and easy manual oversight.
• Pre-product-market-fit; monitoring infrastructure for a feature you might kill is wasted.
• The product processes under 1M tokens per day and the team can manually spot-check.

In these cases, basic logging plus weekly cost review is enough. Build full monitoring when scale or criticality justifies it.

What to Ask Your Engineering Team

1. Can you show me the cost per feature, per tenant for last month? If not, fix this first.
2. What's our p99 latency by endpoint? Averages don't matter; tail does.
3. How are we sampling production for quality? Frequency, sample size, grading method.
4. What alerts fire what threshold trigger what action? Specific runbooks.
5. What's our fallback chain engagement? When does the secondary provider get hit?
6. How do we detect drift? Specific monitoring; not "we'd notice."
7. What's the audit log story? For compliance, every request must be inspectable.

Cost & Timeline Quick Reference

Realistic ranges for monitoring infrastructure:

The investment pays back the first time it catches a regression or cost surprise. Most teams find that within the first 30 days.

The Bottom Line

Production LLM monitoring is required, not optional. Five layers: cost, latency, quality, reliability, safety.

Per-feature cost attribution is the highest-value single investment. Quality drift detection is the most often-skipped. Tail latency monitoring catches UX issues averages hide.

Don't ship AI features without monitoring. The cost of building monitoring late is much higher than building it day-one.