Document Ingestion Pipeline for Enterprise Knowledge Bases

Enterprise knowledge bases contain documents in dozens of formats. Each requires a different parsing strategy.

PDFs (the most common and most challenging):

• Text-based PDFs: Use PyPDF2 or pdfplumber for basic extraction. For layout-aware parsing that preserves tables and columns, use Unstructured.io or Amazon Textract.
• Scanned PDFs: Require OCR first. Tesseract handles English well. For Hindi and regional languages, Google Cloud Vision API delivers 95%+ accuracy.
• PDFs with tables: Apache Tika or Camelot extract tables into structured data. Critical for financial reports and compliance documents.

Microsoft Office documents:

• Word (.docx): python-docx preserves headings, lists, and basic formatting. Heading hierarchy is valuable metadata for chunking.
• PowerPoint (.pptx): python-pptx extracts slide text and speaker notes. Slide titles become section headers for better retrieval context.
• Excel (.xlsx): openpyxl extracts data, but the challenge is contextualizing numbers. We prepend column headers and sheet names as metadata.

Web-based sources:

• Confluence/Notion: API-based extraction preserves page hierarchy and links. Export as HTML, then parse with BeautifulSoup.
• Slack/Teams messages: Thread context matters. Export threads as units rather than individual messages.

Pro tip: Always preserve document metadata (filename, author, creation date, department) during parsing. This metadata powers access control and helps the LLM cite sources accurately.

Chunking — splitting documents into retrieval-sized pieces — directly determines whether your AI gives precise answers or vague summaries.

Fixed-size chunking (simple but limited):

Split text every 500-1000 tokens with 100-200 token overlap. Easy to implement, works reasonably well for homogeneous documents. Problem: it splits mid-paragraph, mid-table, and mid-argument, destroying context.

Semantic chunking (recommended for most use cases):

Split at natural boundaries — headings, paragraph breaks, topic changes. Use heading hierarchy to maintain section context. A chunk from "Section 3.2: Leave Policy for Contract Employees" carries its section title as context, dramatically improving retrieval relevance.

Recursive chunking:

Start with the largest meaningful unit (a full section), then recursively split only if it exceeds the token limit. Preserves context better than fixed-size while staying within embedding model limits.

Agentic chunking (advanced):

Use an LLM to analyze document structure and create semantically meaningful chunks. More expensive (requires an LLM call per document) but produces the highest quality chunks. We use this for high-value documents like legal contracts and compliance manuals.

Our recommendation: Semantic chunking with heading-based boundaries for 80% of documents. Agentic chunking for high-value documents where retrieval accuracy is critical. Target chunk size of 300-500 tokens for text-embedding-3-large, 200-400 tokens for Cohere embed-v3.

Once documents are chunked, each chunk needs to be converted into a vector embedding for storage and retrieval.

Batch processing architecture:

For initial ingestion of large document sets (10K+ documents), batch processing is essential. Use a job queue (Redis Queue or Celery) to manage embedding generation. Rate limiting against embedding API quotas prevents 429 errors.

Embedding pipeline flow:

1. Document parsed and chunked → stored in PostgreSQL with metadata
2. Chunk added to embedding job queue
3. Worker picks up batch of 50-100 chunks
4. Calls embedding API (OpenAI, Cohere, or local model)
5. Vectors stored in vector database with chunk ID reference
6. Status updated in PostgreSQL

Processing speed benchmarks:

• OpenAI text-embedding-3-large: ~3000 chunks/minute (API rate limited)
• Cohere embed-v3: ~2000 chunks/minute
• Self-hosted BGE-M3 on A10G GPU: ~5000 chunks/minute (no rate limits)

For 100K documents (approximately 2M chunks after splitting), expect:

• OpenAI: ~11 hours, cost approximately Rs 15,000
• Cohere: ~16 hours, cost approximately Rs 12,000
• Self-hosted: ~7 hours, infrastructure cost approximately Rs 2,000

The initial ingestion is just the beginning. Enterprise documents change constantly — new SOPs are published, policies are updated, product documentation evolves.

Change detection strategies:

• File system watching: Monitor shared drives and document repositories for new or modified files. Use file hash comparison to detect changes.
• API polling: For Confluence, Notion, and SharePoint — poll the API for recently modified pages. Most APIs support "modified since" queries.
• Webhook integration: Where available (Confluence, Notion), register webhooks for real-time notifications when documents change.

Update pipeline:

When a document changes, the system must: re-parse the document, re-chunk it, generate new embeddings, and replace the old vectors in the database. This must happen atomically — users should never see partial updates where some chunks are old and some are new.

Deletion handling: When a document is archived or deleted, all associated chunks and vectors must be removed from the vector database. We tag each vector with its source document ID, making bulk deletion straightforward.

Recommended update frequency: Real-time for critical documents (HR policies, compliance docs). Daily batch for general documentation. Weekly for archived/reference material.

At enterprise scale, the ingestion pipeline needs to handle tens of thousands of documents reliably.

Infrastructure for scale:

• Queue-based processing: Redis Queue or RabbitMQ ensures no document is lost if a worker crashes. Failed jobs are automatically retried.
• Parallel workers: Run 4-8 workers for parsing and 2-4 for embedding generation. Parsing is CPU-bound; embedding generation is API-rate-limited.
• Monitoring: Track documents processed, chunks generated, embedding failures, and vector database size. Alert on processing backlogs.

Common failure modes and mitigations:

• Corrupted PDFs: Catch parsing exceptions, log the filename, skip and continue. Flag for manual review.
• Encoding issues: Indian documents often mix UTF-8 with legacy encodings. Detect encoding with chardet before parsing.
• Duplicate content: Hash each chunk to detect duplicates across documents. Store only unique chunks to avoid retrieval noise.
• Rate limiting: Implement exponential backoff for embedding API calls. Queue overflow handling prevents memory issues.

Testing retrieval quality:

After ingestion, evaluate retrieval quality by testing 50-100 representative queries against expected source documents. Target 85%+ retrieval accuracy (correct source document in top 5 results) before going live. We maintain evaluation datasets for each client and run quality checks after every major ingestion update.

Boolean & Beyond has built document ingestion pipelines processing 500K+ documents for Indian enterprises. We handle the messy reality of enterprise data — mixed-language documents, legacy formats, compliance requirements, and government forms with specific layouts.

Our production pipelines achieve 90%+ parsing accuracy across document types, with intelligent chunking that preserves context for high-quality retrieval. We include monitoring, error handling, and incremental update capabilities from day one — not as afterthoughts.

Typical pipeline deployment takes 2-3 weeks from document audit to production, with ongoing optimization as your document corpus grows. Contact Boolean & Beyond to discuss your document ingestion requirements.

Explore more from our AI solutions library:

• RAG Implementation & Vector Databases — Choose the right vector database and chunking strategy for your document ingestion pipeline.
• Building Custom MCP Servers: A Developer Guide — Connect your knowledge base to AI assistants via Model Context Protocol for real-time document access.