RAG Pipeline Development

The RAG Pipeline as Production Infrastructure

A RAG demo and a RAG pipeline in production are different categories of system. The demo embeds a corpus once, queries it, and shows a clever answer. The production pipeline ingests new and updated content continuously, embeds it consistently, indexes it durably, hides deleted content immediately, and surfaces failures to operators before users notice.

We help engineering teams build the production-grade pipeline — the part that makes RAG actually work for months and years in production, not just in a demo.

Document Ingestion: Sources and Parsing

Real corpora live in messy places. A representative production RAG ingestion pipeline reads from:

• Document stores — SharePoint, Google Drive, Dropbox, Confluence, Notion, internal CMSes.
• Databases — PostgreSQL tables, document stores, semi-structured JSON.
• APIs and webhooks — ticketing systems, CRMs, helpdesks, internal tools.
• File systems — network shares, S3, GCS, on-premise storage.
• Email and chat — when those are part of the knowledge base.

Each source has its own authentication, rate limits, change-tracking semantics, and access control rules. None of them are "just call an API once."

Parsing each document type requires the right tool:

• Unstructured.io, LlamaParse, Docling — structured parsing of PDFs, Word, PowerPoint with layout awareness (headings, tables, figures).
• Custom parsers — for proprietary formats or domain-specific structure.
• HTML parsing — readability libraries to strip nav and boilerplate.
• OCR — for scanned documents. Tesseract for low-stakes; managed (AWS Textract, Google Document AI) for production-grade extraction.

We invest in this layer because what you parse becomes what you embed becomes what your retriever returns. Garbage at ingest is garbage forever.

Incremental Sync vs Full Reindex

A pipeline that re-ingests the entire corpus on every run cannot run frequently. A pipeline that doesn't re-ingest at all becomes stale and stops being useful.

Production RAG pipelines run incremental sync:

• Source change detection — webhook-based when the source supports it; polling with last-modified timestamps when it doesn't; checksum comparison as a fallback.
• Per-document change handling — new, updated, deleted, permission-changed each have a separate code path.
• Idempotent processing — re-running ingest on the same document produces the same result. Critical for crash recovery.
• Periodic full reindex for catching missed changes — typically nightly or weekly, depending on update velocity.

Most production RAG outages we have diagnosed trace back to the sync layer: missed updates, stale ACLs after permission changes, deleted documents still surfacing.

The Embedding Pipeline

Once parsed, content flows through:

1. Cleaning — remove boilerplate, normalize whitespace, strip noisy content.
2. Chunking — by structure, semantics, or fixed size. Choice depends on content type.
3. Embedding — call to the embedding model API or self-hosted encoder.
4. Indexing — vector + metadata into the vector store.

Operational concerns at each step:

• Rate limit handling — embedding APIs have RPM/TPM limits. Bulk ingestion needs backoff, parallelism control, and cost monitoring.
• Embedding model versioning — when you upgrade the embedding model, you must re-embed the entire corpus. Plan for this from the start.
• Idempotent indexing — re-indexing the same chunk produces the same vector ID. Critical for incremental sync.
• Cost ceilings — bulk embedding can run up unexpected bills. We instrument cost telemetry from day one.

Index Hygiene: Deletes, Updates, ACL Changes

The hard part of operating RAG in production is keeping the index honest with the source.

• Deletes must propagate immediately. If a document is deleted from SharePoint at 9am, no user should see chunks of it in retrieval at 9:01am.
• Updates require atomic chunk replacement. A document that grew from 5 chunks to 7 needs the old 5 deleted and the new 7 indexed in a single logical operation.
• ACL changes drive re-indexing. When a user's access to a document is revoked, every chunk indexed for that document must drop the user from its allowed-principals list.
• Re-embedding on model upgrade must run without taking the index offline. Dual-index strategies (write to both old and new while migrating, then cutover) preserve availability.

We design these flows explicitly. Most pipelines we inherit treat them as edge cases.

Pipeline Observability and Failure Recovery

A RAG pipeline you cannot debug becomes a black box that ages badly.

We instrument:

• Per-stage success/failure metrics — sources fetched, documents parsed, chunks embedded, vectors indexed. Per source, per document type.
• Lag dashboards — time from source update to index update, p50 and p99. The most user-visible quality metric.
• Cost telemetry — embedding tokens, API calls, storage. Per source, per document type.
• Failed document inspection — failed parses, failed embeddings, indexing errors. With enough context to diagnose without re-running the pipeline.
• Reprocessing controls — re-embed a single document, reindex a category, full corpus replay. Essential for debugging and migrations.

Production RAG operations look like data engineering operations because that's what they are.

Cost Optimization in Production RAG

RAG cost has three contributing factors:

• Embedding cost at ingest. Mostly a function of corpus size and update frequency.
• Retrieval cost at query time. Vector store reads, plus reranker calls if used.
• Generation cost at query time. The LLM call dominates total cost for most production RAG.

Optimizations we apply:

• Hierarchical embedding — embed at multiple granularities (paragraph + section), retrieve at the appropriate level.
• Quantization — use binary or scalar quantization on stored vectors when recall budget allows.
• Embedding model tier selection — smaller embedding models for high-volume ingestion, larger models only where they earn it.
• Context budget tuning — fewer retrieved chunks at smaller token counts often produces equal or better answers at lower cost.
• Caching — repeated queries hit a cache before the full pipeline.

We instrument cost per useful response, not just per query. Optimizations that reduce per-query cost but degrade quality often fail this test.

How We Build RAG Pipelines

For most engagements, RAG pipeline engagements typically run 6–12 weeks:

• Weeks 1–2: Source inventory and architecture. What sources, what update velocity, what access control rules, what latency requirements.
• Weeks 3–6: Pipeline implementation. Source connectors, parsing, chunking, embedding, indexing. With observability instrumented from day one.
• Weeks 7–9: Index hygiene and operations. Delete propagation, update flows, ACL sync, dual-index migration patterns.
• Weeks 10–12: Hardening and handoff. Load testing on full corpus, runbooks, retraining materials.

The deliverable is a pipeline the client team operates after the engagement. We invest in observability and runbooks because RAG pipelines that nobody can operate get replaced.

Summary: RAG Pipeline Implementation Stack

1. Treat the pipeline as data engineering, not a script. That's what it is.
2. Invest in parsing quality. Garbage at ingest is garbage forever.
3. Build incremental sync, not periodic full reindex. Full reindex doesn't scale beyond demos.
4. Plan for index hygiene — deletes, updates, ACL changes — as first-class concerns, not edge cases.
5. Plan for embedding model upgrades. You will upgrade; have the dual-index migration ready.
6. Instrument observability and reprocessing controls before launch.
7. Optimize cost at the right layer. Generation cost usually dominates; embedding cost is rarely the bottleneck.

A RAG pipeline that works in demo and survives in production looks structurally different. Most of the work is in the pipeline you don't see in the demo.