Choosing an Embedding Model for Enterprise Search and RAG
Embedding model choice is the single most under-scrutinized decision in most enterprise RAG deployments, and it quietly determines a retrieval system's accuracy ceiling more than the generation model that sits downstream of it. A weak or poorly matched embedding model produces mediocre retrieval no matter how good the LLM answering from those retrieved chunks is, because the model can only reason over what retrieval actually surfaced. Three decisions matter most: embedding dimension and its tradeoff against storage and search speed, multilingual coverage if your documents span languages, and whether the general-purpose embedding model needs domain adaptation for your specific vocabulary, whether that is aerospace engineering terminology, legal contract language, or manufacturing part nomenclature.
Dimension Size: The Storage and Speed Tradeoff
Higher-dimensional embeddings, typically 1024 to 4096 dimensions in current models, capture more semantic nuance and generally retrieve more accurately, but they cost proportionally more in vector database storage and search latency at scale. Many current embedding models support Matryoshka representation learning, which lets you truncate a high-dimensional embedding to a smaller size, say from 1024 down to 256, with a graceful and often modest accuracy loss rather than a cliff, giving you a practical knob to tune the storage-versus-accuracy tradeoff without switching models entirely. For most enterprise document corpora under a few million chunks, the accuracy gain from the largest available dimension usually outweighs the storage cost, but at genuinely large scale, tens of millions of chunks, truncated or lower-dimensional embeddings often make more sense.
- Higher dimensions (1024-4096): better semantic capture, higher storage and search cost at scale
- Matryoshka-capable models: truncate dimensions with graceful, not cliff-edge, accuracy loss
- Under a few million chunks: prioritize accuracy, use the largest practical dimension
- Tens of millions of chunks: consider truncated or smaller-dimension models to control cost and latency
Multilingual Coverage: Do Not Assume English-Only Models Generalize
If your document corpus includes multiple languages, supplier communications, international standards documents, or multilingual customer support tickets, verify that your embedding model was explicitly trained for multilingual retrieval rather than assuming a strong English-language model transfers well. Cross-lingual retrieval, where a query in one language should match a document in another, is a harder capability than simply supporting multiple languages independently, and not every multilingual model handles it well. Test retrieval quality separately for each language pair you actually need, since a model's aggregate multilingual benchmark score can hide meaningfully worse performance on a specific language your corpus depends on, particularly for less commonly benchmarked languages.
Domain Adaptation: When General Embeddings Fall Short
General-purpose embedding models trained primarily on web text and common document types often underperform on specialized enterprise vocabulary, aerospace part specifications, legal boilerplate with precise term-of-art distinctions, or manufacturing process terminology where two superficially similar phrases mean substantively different things. The tell is a retrieval system that returns plausible-looking but wrong documents, chunks that share surface vocabulary with the query but miss the actual intent. Domain adaptation, fine-tuning the embedding model on pairs of your own queries and correct-match documents, typically using contrastive learning approaches, closes this gap meaningfully and is a comparatively cheap fine-tune, often requiring far less data and compute than fine-tuning a generation model.
- Symptom of the gap: retrieval returns surface-similar but semantically wrong documents
- Fix: contrastive fine-tuning on your own query-to-correct-document pairs
- Cheaper than generation model fine-tuning, often effective with a few thousand labeled pairs
- Common domains needing adaptation: engineering specs, legal contract language, regulatory and compliance text
Evaluating Embedding Models the Right Way
Do not evaluate an embedding model in isolation from your actual retrieval pipeline and chunking strategy, since the same embedding model can perform very differently depending on chunk size, overlap, and whether you are using pure vector search or a hybrid approach combining vector and keyword search. Build a retrieval evaluation set from real queries your users actually ask, paired with the documents that should be retrieved for each, and measure recall at various result-set sizes rather than a single top-1 accuracy number. Public embedding leaderboards like MTEB are a reasonable starting filter but suffer from the same contamination and generalization concerns as any other public benchmark, so validate the finalists against your own retrieval evaluation set before committing.
How Netray Selects and Adapts Embedding Models
Netray treats embedding selection as its own evaluation project rather than a default choice bundled into a RAG build, because we have seen more retrieval quality problems traced back to embedding mismatch than to the generation model. We build a retrieval evaluation set from your real queries and documents, test dimension and multilingual fit against your actual corpus, and run domain-adaptation fine-tuning where a general-purpose model's vocabulary gap is measurably hurting recall. For on-premises RAG deployments, we also handle the vector database sizing that follows from your dimension choice, since that decision has real infrastructure cost implications that should be made deliberately rather than inherited from a default configuration.
Frequently Asked Questions
How much do embedding dimensions matter for RAG retrieval accuracy?
They matter meaningfully but with diminishing returns. Higher-dimensional embeddings (1024 to 4096) generally retrieve more accurately but cost more in storage and search latency at scale. Many current models support Matryoshka representation learning, letting you truncate dimensions with graceful accuracy loss rather than a cliff. For most corpora under a few million chunks, prioritize accuracy with a larger dimension; at tens of millions of chunks, consider truncated or smaller models.
Do we need a different embedding model for multilingual documents?
Verify your model was explicitly trained for multilingual and cross-lingual retrieval rather than assuming an English-strong model transfers. Cross-lingual retrieval, matching a query in one language to a document in another, is harder than simply supporting several languages independently, and aggregate multilingual benchmark scores can hide weak performance on specific language pairs your corpus actually needs. Test each language pair separately before committing.
When should we fine-tune an embedding model instead of using a general-purpose one?
When retrieval returns documents that share surface vocabulary with a query but miss its actual intent, a common symptom in specialized domains like aerospace engineering specs, legal contract language, or manufacturing terminology. Domain adaptation through contrastive fine-tuning on your own query-to-document pairs closes this gap and is comparatively cheap, often effective with a few thousand labeled pairs, far less data than fine-tuning a generation model requires.
Key Takeaways
- 1Dimension Size: The Storage and Speed Tradeoff: Higher-dimensional embeddings, typically 1024 to 4096 dimensions in current models, capture more semantic nuance and generally retrieve more accurately, but they cost proportionally more in vector database storage and search latency at scale. Many current embedding models support Matryoshka representation learning, which lets you truncate a high-dimensional embedding to a smaller size, say from 1024 down to 256, with a graceful and often modest accuracy loss rather than a cliff, giving you a practical knob to tune the storage-versus-accuracy tradeoff without switching models entirely.
- 2Multilingual Coverage: Do Not Assume English-Only Models Generalize: If your document corpus includes multiple languages, supplier communications, international standards documents, or multilingual customer support tickets, verify that your embedding model was explicitly trained for multilingual retrieval rather than assuming a strong English-language model transfers well. Cross-lingual retrieval, where a query in one language should match a document in another, is a harder capability than simply supporting multiple languages independently, and not every multilingual model handles it well.
- 3Domain Adaptation: When General Embeddings Fall Short: General-purpose embedding models trained primarily on web text and common document types often underperform on specialized enterprise vocabulary, aerospace part specifications, legal boilerplate with precise term-of-art distinctions, or manufacturing process terminology where two superficially similar phrases mean substantively different things. The tell is a retrieval system that returns plausible-looking but wrong documents, chunks that share surface vocabulary with the query but miss the actual intent.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
RAG vs Fine-Tuning Decision Assessment
Answer 8 questions about knowledge volatility, citation needs, data availability, and team capability to find whether RAG, fine-tuning, or a hybrid fits your project.
Free ToolPrivate RAG Corpus Sizing Calculator
Estimate chunk counts, vector index storage, raw text volume, and embedding compute time before you build a private retrieval system over your document estate.
Free ToolRAG Chunking Strategy Calculator
Turn corpus size, chunk length, and overlap into a concrete chunk count, embedding cost, and vector storage footprint before you build the ingestion pipeline.
Terms used in this article
Is your RAG system's retrieval quality the bottleneck, not the LLM? Netray will evaluate embedding models against your real queries and documents, including domain adaptation where it matters.
Related Resources
Reading LLM Benchmarks Skeptically: A Practitioner's Guide
Interpret LLM benchmarks skeptically: contamination risk, private evals, and why leaderboard rank should never be your only model selection signal.
AI & AutomationChoosing a Vector Database for On-Prem RAG
Compare vector databases for on-prem RAG: Qdrant, Weaviate, Milvus, and pgvector on performance, filtering, security, and ops burden for enterprise AI.
AI & AutomationSmall Language Models for Enterprise: When Smaller Wins
When small language models like Phi-4, Gemma 3, and Qwen3 small variants beat large models on cost, latency, and task-specific enterprise accuracy.