AI & Automation5 min readNetray Engineering TeamUpdated March 4, 2025

Choosing a Vector Database for On-Prem RAG: Qdrant vs Weaviate vs Milvus vs pgvector

Choosing a vector database for on-prem RAG comes down to four candidates for most enterprises: Qdrant, Weaviate, Milvus, and pgvector - all open source, all deployable entirely inside your firewall. The right choice depends less on raw benchmark numbers than on metadata filtering strength, operational burden, and how the database enforces access control, because enterprise RAG queries are always filtered queries: by site, document class, permission group, and classification tier. This guide compares the four on the criteria that actually decide manufacturing deployments, including CMMC-relevant security features and realistic scale thresholds.

The Four Realistic On-Prem Candidates

Qdrant (Rust-based, Apache 2.0) is the current default for mid-size deployments: fast filtered HNSW search, first-class payload indexing, simple single-binary or Docker deployment, and built-in API-key auth with RBAC in recent releases. Weaviate adds a GraphQL API, built-in hybrid (BM25 + vector) search, and modular embedding integrations - attractive when teams want batteries included. Milvus is the scale play: billions of vectors, GPU-accelerated indexing, but a multi-component architecture (etcd, Pulsar/Kafka, MinIO) that demands real Kubernetes operations maturity. pgvector turns PostgreSQL 15+ into a capable vector store - the right answer when your corpus is under a few million chunks and your DBA team already runs Postgres with backups, HA, and audit logging solved.

Qdrant: best filtered-search performance per ops dollar; single binary; ideal for 1M-100M vector deployments
Weaviate: built-in hybrid BM25+vector search and reranking modules; heavier memory footprint
Milvus: billion-scale and GPU indexing, but 5+ infrastructure components to operate and patch
pgvector: zero new infrastructure if you run Postgres; solid to low-millions of vectors with HNSW indexes

Metadata Filtering Is the Real Benchmark

Published ANN benchmarks measure unfiltered nearest-neighbor speed, but production ERP queries look like: 'similar nonconformance reports, site = Plant 2, item class = machined, classification below Controlled, user in group X.' Filtered search behavior varies sharply. Qdrant filters during HNSW graph traversal, keeping recall high even when filters eliminate 95% of candidates. pgvector with HNSW historically struggled with heavy filters (post-filtering can return too few results), though iterative scan options in pgvector 0.8+ largely fix this. Weaviate and Milvus both handle filtered search well at moderate selectivity. Test with your real filter distributions: build a 100-query eval from actual user questions, measure recall@10 against a ground-truth set, and require above 90% before committing.

Security and Compliance Features That Matter On-Prem

For CUI-adjacent deployments, the vector database sits inside your CMMC boundary and must support the same controls as any CUI system. Check for: TLS on all endpoints, API-key or OIDC authentication, role-based access scoped per collection, and encryption at rest (native or via LUKS/dm-crypt on the volume). Qdrant and Weaviate ship RBAC in current open-source or lightly licensed tiers; Milvus offers RBAC with users and privileges; pgvector inherits PostgreSQL's mature roles, row-level security, and pgaudit - the strongest story if you need row-level permission trimming enforced in the database itself. Whichever you pick, put the database on the same isolated VLAN as inference nodes, deny egress, and include it in your SSP asset inventory - assessors increasingly ask where embeddings of CUI documents live.

Require TLS, authentication, and per-collection RBAC as table stakes for any CUI-adjacent deployment
pgvector inherits PostgreSQL row-level security and pgaudit - strongest native permission-trimming story
Encrypt at rest via native support or LUKS volumes, and document it against SC.L2-3.13.16
List the vector database in your SSP asset inventory; embeddings of CUI are still CUI-derived data

Sizing, Cost, and a Practical Decision Rule

Scale planning is simpler than vendors imply. A 1536-dimension embedding with metadata costs roughly 8-10 KB in memory with HNSW; 5 million chunks (a substantial manufacturing document corpus - tens of thousands of documents) fits in about 50 GB of RAM on a single Qdrant or pgvector node costing under $10,000. The practical decision rule: under ~3 million chunks and an existing Postgres team, use pgvector; 3 million to 100 million chunks, use Qdrant for filtered-search performance with minimal ops; genuine multi-billion-vector ambitions with a Kubernetes platform team, consider Milvus; want built-in hybrid search without assembling components, Weaviate. Migration between them is a re-embedding-free export/import of vectors and payloads, so the cost of a wrong first choice is days, not months - do not over-engineer round one.

How Netray Builds Retrieval Layers That Hold Up

Netray deploys the retrieval layer as part of every on-prem RAG engagement: database selection benchmarked on your actual documents and filter patterns, ingestion pipelines from SyteLine, Infor LN, SharePoint, and PLM systems, and permission-trimmed retrieval synced to Entra ID groups. We deliver a measured recall report - typically above 90% recall@10 on client eval sets - rather than vendor benchmark screenshots. Netray AI agents then use this layer for quoting, nonconformance analysis, and documentation tasks. A typical retrieval layer for a mid-size manufacturer deploys in 2-3 weeks on a single server, and clients avoid the 6-12 month science project that stalls most internal RAG efforts.

Frequently Asked Questions

Which vector database is best for on-premise RAG?

For most enterprises, Qdrant is the strongest default: excellent filtered-search performance, simple single-binary deployment, and RBAC support, covering 1 million to 100 million vectors on one node. Use pgvector if you already operate PostgreSQL and your corpus is under a few million chunks - you inherit mature backups, HA, and row-level security. Choose Milvus only for billion-scale needs with a real Kubernetes team, and Weaviate when built-in hybrid BM25+vector search matters most.

Can PostgreSQL with pgvector replace a dedicated vector database?

Yes, up to a few million vector chunks - which covers many enterprise document corpora. pgvector 0.7+ with HNSW indexes delivers competitive latency, and PostgreSQL brings row-level security, pgaudit, point-in-time recovery, and a team that already knows how to run it. Its weakness is heavily filtered searches at larger scale, improved by iterative scans in 0.8+. If your corpus will exceed roughly 3-5 million chunks or filters are highly selective, a dedicated engine like Qdrant is safer.

How much hardware does a vector database need for enterprise RAG?

Less than most teams expect. At 1536 dimensions with HNSW indexing, budget roughly 8-10 KB of RAM per chunk: 5 million chunks - tens of thousands of documents - needs about 50 GB of RAM plus CPU, no GPU required. A single server under $10,000 handles retrieval for most mid-size manufacturers with sub-50 ms query latency. Scale-out clustering only becomes necessary in the hundreds of millions of vectors.

Key Takeaways

1The Four Realistic On-Prem Candidates: Qdrant (Rust-based, Apache 2.0) is the current default for mid-size deployments: fast filtered HNSW search, first-class payload indexing, simple single-binary or Docker deployment, and built-in API-key auth with RBAC in recent releases. Weaviate adds a GraphQL API, built-in hybrid (BM25 + vector) search, and modular embedding integrations - attractive when teams want batteries included.
2Metadata Filtering Is the Real Benchmark: Published ANN benchmarks measure unfiltered nearest-neighbor speed, but production ERP queries look like: 'similar nonconformance reports, site = Plant 2, item class = machined, classification below Controlled, user in group X.' Filtered search behavior varies sharply. Qdrant filters during HNSW graph traversal, keeping recall high even when filters eliminate 95% of candidates.
3Security and Compliance Features That Matter On-Prem: For CUI-adjacent deployments, the vector database sits inside your CMMC boundary and must support the same controls as any CUI system. Check for: TLS on all endpoints, API-key or OIDC authentication, role-based access scoped per collection, and encryption at rest (native or via LUKS/dm-crypt on the volume).

Put this into numbers

Free interactive tools for exactly this problem. No signup to use them.

Free Tool

AI Model Selection Assessment

Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.

Free Tool