Model Quantization for Production Deployment

If You Remember Nothing Else

Quantization is production table stakes. Every serious LLM deployment uses it. The question isn't whether to quantize, it's which method and at what precision.

The right defaults: AWQ INT4 for self-hosted GPU deployments (60 to 75% cost reduction at minimal quality loss), FP8 on H100 or Ada Lovelace hardware where supported (40 to 60% cost reduction), GGUF for CPU and edge (the llama.cpp ecosystem default).

The rule that matters: never deploy quantization without validating quality on a representative test set. Different methods, different models, and different tasks interact in ways that defy generalization. A quantization that loses 0.5 points on benchmark A may lose 5 points on benchmark B for the same model. The cost saving isn't real if the quality loss is unmeasured.

Recommendations by Situation

Worked Examples

Example 1: Cost-driven migration from FP16 to INT4 (mid-volume production)

A SaaS company runs a fine-tuned Llama 70B at FP16 on 2x A100s. Inference cost: about $4,500 per month. They want to reduce cost.

The right approach: AWQ INT4 on a single H100. Quality validation on a 200-example test set showed a 0.7-point drop on their primary metric (acceptable; user impact would be minor based on their threshold). Final inference cost: about $1,200 per month. Saving: $3,300 per month, $40K per year.

What worked: a 1-week validation phase before committing. The team built a held-out test set, ran both FP16 and AWQ INT4 versions, compared outputs, and confirmed no critical regressions on edge cases.

What they nearly got wrong: shipping the quantized model without the test set. A first attempt without validation had shown promise on benchmarks but degraded badly on a specific class of complex multi-step queries. The test set caught it before users would have.

What to remember: validation matters more than method. Even AWQ INT4 (a strong default) has tasks where it underperforms. Always validate on real workload patterns.

Example 2: H100-native deployment with FP8 (latency-critical)

A real-time voice assistant deployment runs on H100 hardware. Latency target: sub-150ms p99.

The right approach: FP8 quantization. H100's native FP8 Tensor Cores deliver 2x the throughput of FP16 with negligible quality loss (validated on a 300-example test set). The latency budget held; user experience didn't change.

What worked: matching the quantization to the hardware. FP8 on H100 isn't just about cost; it's the right precision for the GPU.

What they nearly got wrong: defaulting to AWQ INT4 (the more aggressive option). The quality loss would have been larger, and the throughput gain was smaller because H100's FP8 Tensor Cores are highly optimized.

What to remember: hardware-native quantization (FP8 on H100/Ada) is often the best fit even when more aggressive options exist. The hardware tells you what to use.

Example 3: Edge deployment with GGUF (offline use)

A field operations app runs Llama 3.1 8B on technicians' laptops in remote sites. No cloud connectivity available.

The right approach: GGUF Q4_K_M quantization via llama.cpp. Model fits in about 5GB of RAM, runs at acceptable speed (about 25 tokens per second on Apple Silicon, 15 tokens per second on Windows laptops with GPU).

What worked: GGUF's broad ecosystem support. Ollama, LM Studio, and llama.cpp all use GGUF natively, so deployment was straightforward across platforms.

What they nearly got wrong: trying to use AWQ on edge hardware. AWQ is designed for NVIDIA GPUs; on consumer laptops without dedicated GPUs, it underperforms compared to GGUF.

What to remember: GGUF is the right default for edge and CPU deployments. For server-side GPU inference, AWQ wins; for off-cloud, GGUF wins.

Anti-Patterns to Watch For

"We'll quantize at deploy time, no need to validate"

What it looks like: cost-optimization plans without a quality validation step.

Why it's wrong: quantization quality loss is unpredictable. Some models tolerate INT4 with no measurable loss; others lose 5+ points on the same setup. Without validation, you're shipping unknowns.

How to redirect: require a held-out test set comparison before any quantized model goes to production. The validation typically takes 2 to 5 days; far less than the time to debug post-shipping quality regressions.

"INT4 saves the most, let's go with INT4"

What it looks like: quantization choice driven by aggression rather than appropriateness.

Why it's wrong: INT4 has the largest quality risk. Some workloads handle it perfectly; others lose meaningfully. Going aggressive without measuring is a coin flip.

How to redirect: start with INT8 as a baseline. Move to INT4 only when measured quality holds up. Skip the gamble.

"Quantization is one-and-done"

What it looks like: quantization treated as a deployment step rather than an ongoing concern.

Why it's wrong: when you fine-tune the model further, the quantization needs to be redone. When you upgrade the base model, quality validation needs to rerun. Quantization is part of the model lifecycle.

How to redirect: bake quantization validation into the CI/CD pipeline. Every model change triggers a quantization re-validation, not just the initial deployment.

"Let's use the latest quantization method"

What it looks like: chasing newer methods (like 2-bit, ternary, or experimental schemes).

Why it's wrong: experimental quantization methods often work in published benchmarks but fail in production. The ecosystem support is thin; debugging is hard.

How to redirect: stick with mature methods (AWQ, GPTQ, FP8, GGUF) for production. Treat experimental methods as research projects, not deployment plans.

"We'll quantize after deployment if cost becomes an issue"

What it looks like: deferring quantization until a cost crisis forces it.

Why it's wrong: retrofitting quantization on a production system is harder than designing for it. Caching, monitoring, and serving infrastructure all assume specific model sizes.

How to redirect: quantize from day one of production deployment. The cost savings start immediately, the engineering work is the same regardless of when you do it.

When NOT to Quantize

Specific cases where the answer is don't quantize:

• The model is too small to matter. A 1B parameter model in FP16 is 2GB; quantization saves maybe 1GB. Probably not worth the validation work.
• The workload is extremely quality-sensitive (medical decisions, legal advice, financial recommendations) and even small quality losses are unacceptable. Validate FP16 first; quantize only if cost forces it.
• The model is brand-new and the quantization libraries don't yet support it well. Wait 1 to 3 months for tooling to catch up.
• You have no evaluation infrastructure. Quantization without validation is risk; build the evaluation harness first.
• The base model changes weekly. Quantization has to be redone each time; the operational overhead may not justify the savings.

What to Ask Your Engineering Team

1. Which quantization method are you proposing, and why this one for our model?
2. What's the projected cost savings vs the FP16 baseline?
3. What's the validation plan? Specific test set, specific quality metrics.
4. What's the worst-case quality loss tolerance? Specific number, not "minimal."
5. How will we know if quantization quality has regressed in production?
6. What hardware is this deployed on? FP8 only works on H100/Ada; INT4 has different optimal paths on different GPUs.
7. What's the rollback plan if quality drops in production?
8. Is the validation test set locked, or can it move?

Cost & Timeline Quick Reference

Realistic ranges for quantization deployment:

Validation is the longest part. Plan for 1 week including test set construction.

The Bottom Line

Quantization is required for cost-effective production. AWQ INT4 is the right default for GPU deployments. FP8 on H100/Ada when supported. GGUF for CPU and edge.

Always validate quality on a held-out test set. Always. The cost savings are real but unmeasured quality loss costs more than the savings save. Engineering teams that skip validation tend to ship quantization that silently degrades user experience until someone notices.

Don't let cost optimization ship without measurement.