Most teams default to LangChain without evaluating the abstraction cost. A balanced analysis of when RAG frameworks help, when they add complexity, and the engineering trade-offs that determine which approach ships better products.
Every team building a RAG application hits the same moment. Someone opens a new project, installs LangChain, follows a quickstart guide, and has a working prototype in 45 minutes. It retrieves documents, sends them to an LLM, and generates answers. The demo works. Leadership is impressed. The team commits to building on LangChain.
Six months later, that same team is debugging a LangChain chain that wraps a chain that wraps a retriever that wraps a vector store adapter, and nobody can trace why a specific query returns irrelevant results. The abstraction layers that made the prototype fast now make production debugging slow. This is not a hypothetical scenario. We have seen it repeat across multiple AI teams in Bengaluru building production RAG systems.
LangChain's Python package installs over 50 transitive dependencies and imports a significant portion of them at module load time. A minimal LangChain RAG chain with a vector store retriever and OpenAI LLM consumes approximately 180-220 MB of RAM at idle, compared to 40-60 MB for an equivalent custom implementation using just the openai, httpx, and pgvector libraries. On serverless platforms like AWS Lambda, this translates to 2-4 seconds of cold start time versus under 1 second for the lean equivalent.
The memory overhead compounds in multi-worker deployments. A Gunicorn server with 8 workers running LangChain-based RAG consumes 1.6-1.8 GB of RAM before handling a single request. The same logic implemented without LangChain fits in 400-500 MB across 8 workers. On a cloud instance, that difference is about $80/month, which sounds trivial until you multiply it across staging, production, and preview environments.
A typical LangChain RAG chain involves 6-8 abstraction layers between your application code and the actual API call: your chain, the chain's run method, the prompt template, the retriever, the vector store adapter, the embedding model adapter, and the LLM adapter. When a query returns poor results, you need to inspect intermediate values at each layer. Without LangSmith or verbose logging enabled at initialization, these intermediate values are invisible, making root cause analysis a guessing game.
LangSmith is LangChain's companion observability platform that traces every step of a chain execution: retriever calls, prompt formatting, LLM inputs and outputs, and token usage. It provides a visual trace view that makes debugging multi-step chains significantly easier. The free tier supports 5,000 traces per month, and the paid tier starts at $39/month for 50,000 traces. The catch is vendor coupling: LangSmith only traces LangChain chains, so if you later migrate away from LangChain, you lose your observability investment.
A custom RAG pipeline can instrument each step with OpenTelemetry spans, sending traces to any compatible backend, Jaeger, Grafana Tempo, Datadog, or Honeycomb. Each span captures the retriever query, retrieved document IDs, chunk content, prompt tokens, LLM response, and latency. This approach costs more engineering time upfront, roughly 2-3 days to implement properly, but gives you vendor-agnostic observability that works regardless of which LLM provider or vector database you use. Teams that already run OpenTelemetry for their application services get RAG observability as a natural extension of their existing tooling.
LangChain's default RecursiveCharacterTextSplitter splits documents by trying progressively smaller separators: double newlines, single newlines, spaces, then characters. It works reasonably for prose documents, producing chunks of consistent size with configurable overlap. For a 10-page technical document, it generates 40-60 chunks of 500 tokens each, with 50-token overlap between adjacent chunks.
The problem emerges with structured documents. A legal contract where Section 3.2.1 references definitions in Section 1.4 gets split into chunks that lose this cross-reference context. A codebase where a function definition spans 80 lines gets split mid-function. LangChain provides specialized splitters for code and markdown, but they handle format, not semantics. No character-based splitter understands that a SQL query's WHERE clause is meaningless without the SELECT and FROM that precede it.
Semantic chunking uses embedding similarity between adjacent sentences to find natural topic boundaries. When the cosine similarity between consecutive sentence embeddings drops below a threshold (typically 0.75-0.85), a chunk boundary is inserted. This produces variable-length chunks that align with topic shifts in the document. On our benchmarks with technical documentation, semantic chunking improves retrieval precision by 12-18% compared to recursive character splitting, because each chunk represents a coherent idea rather than an arbitrary window of text.
The highest-quality chunking strategy is document-aware: parsing the document's native structure (HTML headings, PDF sections, markdown headers) and using those structural boundaries as chunk boundaries. A 50-page product manual chunked by its existing section structure produces chunks that are self-contained, properly titled, and contextually meaningful. This approach requires per-format parsers and is harder to implement generically, which is why frameworks default to simpler strategies. But for teams with a known document corpus, investing in document-aware chunking often produces the single largest improvement in RAG retrieval quality.
Multi-query retrieval generates 3-5 paraphrased versions of the user's query using an LLM, runs vector search for each, and merges the results. This compensates for the fragility of single-query embedding: a user asking 'how to fix memory leaks in Node.js' and one asking 'Node.js heap out of memory error resolution' should retrieve the same documents, but their embeddings may differ enough to return different results. Multi-query retrieval costs 3-5x more in embedding API calls and adds 200-400 ms of latency, but improves recall by 8-15% on diverse query patterns.
HyDE asks the LLM to generate a hypothetical answer to the user's question, then uses that answer's embedding (not the question's embedding) to search the vector store. The intuition is that the hypothetical answer's embedding will be closer to the actual answer's embedding than the question's embedding would be. In practice, HyDE improves retrieval for factual questions by 10-20% but can degrade performance for open-ended or creative queries where the LLM's hypothetical answer diverges from what is actually in the corpus.
Self-query retrieval uses an LLM to parse the user's natural language query into a structured filter plus a semantic query. For example, 'show me Python tutorials about async programming published after 2024' becomes a vector search for 'async programming' filtered by language='Python' and published_date > 2024-01-01. LangChain provides a SelfQueryRetriever that handles this pattern. Implementing it custom is straightforward but requires maintaining a schema description that the LLM uses to generate filters, which changes as your metadata evolves.
Vector search retrieves candidates quickly but imprecisely, because bi-encoder embeddings compress entire documents into single vectors, losing nuance. Cross-encoder re-ranking passes each (query, document) pair through a model that attends to both simultaneously, producing a relevance score that captures fine-grained semantic relationships. Cohere Rerank, a popular hosted cross-encoder, processes 1,000 document pairs per second at roughly $1 per 1,000 search queries (assuming 20 candidates per query). The typical pattern is to retrieve 50-100 candidates from the vector store, re-rank with a cross-encoder, and return the top 5-10.
On internal benchmarks with a 500K-document technical knowledge base, adding Cohere Rerank after pgvector retrieval improved answer relevance (measured by human evaluation) from 72% to 89%. The latency cost was 80-120 ms per query for re-ranking 50 candidates. For most RAG applications, re-ranking is the highest-impact improvement you can make after getting basic retrieval working.
The most common production RAG failure is not hallucination, it is poor retrieval. The LLM receives irrelevant context and generates a plausible-sounding answer from that irrelevant context. Users report 'the AI is wrong' when the actual bug is in the retriever, not the LLM. Without logging retrieved chunks alongside LLM responses, teams spend weeks tuning prompts when they should be fixing their chunking strategy or embedding model selection.
A retriever that returns 10 chunks of 500 tokens each, plus a system prompt and conversation history, can easily exceed the effective context window where the LLM pays attention. Research shows that LLMs attend most strongly to the beginning and end of context windows, losing information in the middle. The fix is not to increase top_k, it is to decrease chunk size, improve retrieval precision so fewer chunks are needed, or implement a summarization step that compresses retrieved context before sending it to the LLM.
When source documents are updated, the embeddings in the vector store become stale. A product manual updated with a new version number still has old embeddings reflecting the previous version's content. LangChain does not handle this automatically. Custom pipelines can implement content hashing, re-embedding only chunks whose source content has changed, which reduces re-embedding costs by 80-90% on typical knowledge bases where most content is stable between updates.
At 10,000 queries per day using a retrieval pipeline with embedding search, re-ranking, and Claude as the LLM: the embedding API costs are approximately $2-3/day for query embeddings, re-ranking adds $10/day with Cohere Rerank, and LLM inference is $15-30/day depending on response length. Total: roughly $800-1,000/month in API costs. LangChain adds zero cost to this equation since it is open source. The cost difference between LangChain and custom code is purely in engineering time: LangChain saves 2-3 weeks of initial development but can cost 2-4 extra weeks in debugging and customization over six months.
At 100,000 queries per day, the API costs dominate at $8,000-10,000/month, and the LangChain vs custom decision is irrelevant to the cost equation. The decision at this scale is about control: can you implement custom caching (embedding cache hit rates of 30-40% on repeated queries), request batching, and provider fallback logic more easily with or without LangChain's abstractions?
RAGAS evaluates RAG pipelines on four dimensions: faithfulness (does the answer stick to the retrieved context), answer relevance (does the answer address the question), context precision (are retrieved chunks relevant), and context recall (did retrieval find all relevant information). It uses an LLM as a judge to score each dimension. Running RAGAS on a test set of 200 questions costs approximately $5-10 in LLM API calls and takes 15-20 minutes. We run RAGAS evaluations weekly on production RAG systems to catch retrieval degradation before users notice it.
Beyond offline evaluation, production RAG systems benefit from real-time context relevance scoring. After retrieval but before LLM generation, a lightweight classifier scores each retrieved chunk's relevance to the query. Chunks below a threshold are dropped, reducing noise in the LLM's context window. This classifier can be a fine-tuned DistilBERT model that adds less than 10 ms of latency and reduces irrelevant context by 25-40%, directly improving answer quality. LangChain does not provide this as a built-in pattern, making it a natural extension point for custom pipelines.
Use LangChain when you are prototyping and need to validate a RAG concept in under a week, when your team is new to LLM application development and benefits from LangChain's opinionated patterns, when your use case is a standard question-answering system over a document corpus without complex retrieval logic, or when you plan to use LangSmith for observability and accept the vendor coupling trade-off.
Build custom when your RAG pipeline requires domain-specific chunking strategies that LangChain's splitters cannot handle, when you need fine-grained control over retrieval (custom re-ranking, hybrid search, self-query with domain-specific filters), when you are deploying on serverless and need to minimize cold start and memory footprint, when your team has experience building production APIs and prefers explicit code over framework abstractions, or when you are building for scale beyond 50K queries per day and need custom caching, batching, and fallback logic.
A pragmatic approach we have seen work well for teams in Bengaluru: prototype with LangChain to validate the concept and get buy-in, then rewrite the retrieval and generation pipeline in custom code once you understand your specific requirements. The prototype takes 1-2 weeks. The custom rewrite takes 3-4 weeks. But the custom version is easier to debug, cheaper to run, and faster to iterate on for the next two years of the product's life.
Boolean and Beyond Team
Insight → Execution
Book an architecture call, validate cost assumptions, and move from strategy to production with measurable milestones.
LangChain remains the most popular RAG framework with an active ecosystem, but alternatives like LlamaIndex (for data-focused RAG), Haystack (for production pipelines), and direct SDK usage have grown significantly. LangChain's value is highest for teams building standard RAG patterns quickly. For production systems with custom retrieval logic, the trend is toward lightweight libraries or custom code with OpenTelemetry-based observability.
At 10,000 queries per day with embedding search, re-ranking, and a frontier LLM, expect $800-1,200/month in API costs plus $200-400/month for vector database infrastructure. The LLM inference cost dominates at roughly 60-70% of total spend. Costs scale linearly with query volume. Caching repeated queries and embedding results can reduce API costs by 20-40% depending on query diversity.
Re-ranking. Adding a cross-encoder re-ranker (Cohere Rerank or a self-hosted cross-encoder model) between vector retrieval and LLM generation typically improves answer relevance by 15-25% measured by human evaluation. It costs $1 per 1,000 queries with Cohere and adds 80-120 ms of latency. After re-ranking, the next biggest improvement is chunking strategy, specifically moving from fixed-size character splitting to semantic or document-aware chunking.
LangChain runs in production at many companies. The question is not whether it can, but whether the abstraction overhead is worth it for your specific use case. For standard question-answering over documents with LangSmith observability, LangChain in production is fine. For complex retrieval pipelines with custom re-ranking, hybrid search, caching, and provider fallback, the framework's abstractions become obstacles rather than accelerators.
Use the RAGAS framework to score faithfulness, answer relevance, context precision, and context recall on a test set of 200+ questions with known good answers. Run this evaluation weekly against production data. Additionally, log every retrieved chunk alongside LLM responses so you can audit retrieval quality when users report bad answers. The most common production RAG failure is poor retrieval misdiagnosed as an LLM problem.
If you commit to LangChain long-term, LangSmith provides excellent trace visualization with minimal setup. If you want vendor-agnostic observability or already use OpenTelemetry, build custom instrumentation. The implementation cost is 2-3 days of engineering time. Custom observability also captures metrics LangSmith does not, such as embedding cache hit rates, re-ranker score distributions, and retrieval latency by document source.
RAG and knowledge retrieval systems
Build enterprise RAG systems with vector databases, intelligent chunking, and secure deployment. Production-ready retrieval-augmented generation for knowledge bases, customer support, and document processing.
Learn moreDocument ingestion, embedding, and retrieval at production grade
Production-grade RAG pipelines built for performance, maintainability, and your specific retrieval requirements. We design, build, and optimize retrieval-augmented generation systems, from document ingestion and embedding to custom retrieval logic and LLM integration, without unnecessary framework overhead.
Learn moreExplore related services, insights, case studies, and planning tools for your next implementation step.
Delivery available from Bengaluru and Coimbatore teams, with remote implementation across India.