Choosing a Vector Database for RAG

Choosing a Vector Database for RAG

The vector database is the core storage and retrieval layer of any RAG system. Choosing the wrong one can lead to performance bottlenecks, excessive costs, or operational complexity that impedes your team. The right choice depends on five factors: dataset scale, query latency requirements, deployment preference (managed vs. self-hosted), filtering needs, and whether you need hybrid search (vector + keyword).

Managed Cloud Solutions: Pinecone and Zilliz

Pinecone: Simplicity and Performance

Pinecone is fully managed with no infrastructure to operate. It supports namespaces for multi-tenant isolation, metadata filtering, sparse-dense hybrid search (combining BM25 keyword scores with dense vector scores), and serverless and pod-based deployment. Serverless scales to zero cost, pod-based provides consistent low-latency (p99 <10ms). Pricing is consumption-based: roughly $0.096 per 1M read units on serverless. Best for: teams that want to ship quickly without managing infrastructure and can tolerate vendor lock-in.

Zilliz Cloud (Milvus)

Zilliz is the managed cloud offering of open-source Milvus. It handles billion-scale vector workloads, supports multiple index types (HNSW, IVF_FLAT, IVF_PQ, DiskANN), and provides better price-performance at large scale compared to Pinecone. If you need self-hosted, Milvus can run on Kubernetes with Helm charts. The operator model (Milvus Operator) simplifies cluster lifecycle management. Best for: teams with large-scale requirements (>100M vectors) comfortable with more operational complexity.

Self-hosted Options: Weaviate and Qdrant

Weaviate: Hybrid Search and Modules

Weaviate combines vector search (HNSW) with BM25 keyword search natively, and the hybrid search result can be fused using Reciprocal Rank Fusion (RRF) or linear combination. It offers modules for automatic vectorization (text2vec-openai, text2vec-cohere) that call embedding APIs during ingest, simplifying the pipeline. Weaviate Cloud Service (WCS) is the managed offering; self-hosted via Docker Compose or Kubernetes Helm. GraphQL and REST APIs provide flexible querying. Best for: use cases requiring hybrid search where keyword precision matters alongside semantic search.

Qdrant: Filtering Performance and Rust Efficiency

Qdrant is written in Rust, delivering exceptional memory efficiency and throughput. Its filtered HNSW implementation outperforms competitors for queries with strict metadata filters (e.g., "find similar products in category=electronics where price<$100"). It supports binary quantization (reducing memory by 32x vs float32), scalar quantization, and product quantization for cost optimization. Qdrant Cloud offers managed hosting; self-hosted supports Docker and Kubernetes. Payload indexing allows fast filtered searches without post-filtering. Best for: complex metadata filtering, latency-sensitive workloads, or when self-hosting Rust infrastructure is feasible.

PostgreSQL-based: pgvector and pgvectorscale

pgvector: Pragmatic Choice for PostgreSQL Users

pgvector is a PostgreSQL extension adding vector similarity search via HNSW and IVF indexes. If you already use PostgreSQL (or RDS, Cloud SQL, Supabase, Neon), you can add vector search without new infrastructure. Schema: `ALTER TABLE documents ADD COLUMN embedding vector(1536); CREATE INDEX ON documents USING hnsw(embedding vector_cosine_ops);`. Supports exact and approximate nearest neighbor search. Performance degrades above 1M vectors compared to dedicated vector databases. pgvectorscale (Timescale) adds StreamingDiskANN index for better performance at scale. Best for: smaller RAG applications (<1M chunks) already on PostgreSQL who want minimal operational overhead.

Chroma: Development and Prototyping

Chroma is an open-source, developer-friendly vector database designed for fast prototyping. It runs in-process (no separate server) or as a client-server, making it zero-setup for local development. Python-first API is intuitive. Not designed for production workloads at scale—lacks the operational maturity, durability guarantees, and performance of Pinecone/Qdrant/Weaviate. Use Chroma for: local development, proof-of-concepts, tutorials. Migrate to a production-grade solution before deploying at scale.

Decision Framework: How to Choose

Scale and Performance Thresholds

Under 1M vectors: pgvector (if PostgreSQL stack), Chroma (dev only), or Pinecone serverless. 1M to 100M vectors: Pinecone pod-based, Qdrant, Weaviate, or Milvus. Above 100M vectors: Milvus/Zilliz, Qdrant with replication, or Weaviate with custom configuration. Latency requirement p99 <10ms: Pinecone pods, Qdrant (with payload indexing), or Milvus with GPU acceleration. Hybrid search (vector + keyword) required: Weaviate or Qdrant (both support fusion natively).

Operational and Cost Considerations

Total cost includes storage, compute, and engineering time. Managed services (Pinecone, Zilliz) have higher $/GB but zero DevOps cost. Self-hosted (Qdrant, Weaviate, Milvus) requires Kubernetes expertise and ongoing maintenance. For early-stage products, the engineering time saved by managed services typically outweighs the premium pricing. For mature products at scale, self-hosted can reduce costs by 50-70%. Evaluate both the dollar cost and the ops burden honestly before deciding.

Summary: Recommended Defaults

Start with Pinecone serverless for managed simplicity during MVP and early growth. Switch to Qdrant (self-hosted) or Milvus when cost optimization becomes necessary at scale. Use pgvector if you are already on PostgreSQL and dataset size is under 1M vectors. Use Weaviate when hybrid search is a core requirement. Benchmark your specific workload (document count, query patterns, filter selectivity) before committing to any option—published benchmarks rarely reflect real-world conditions precisely.