RAG Evaluation Metrics: How to Know If Your System Actually Works
Most teams evaluate a RAG system by reading a dozen answers and deciding whether they look reasonable, which is not evaluation, it is anecdote. A real RAG evaluation separates retrieval quality from generation quality, because they fail for different reasons and are fixed with different work. Retrieval metrics like recall@k and mean reciprocal rank tell you whether the right chunk was even found. Generation metrics like faithfulness and answer relevance tell you whether the model used what was retrieved and actually answered the question asked. Build both from a golden set of real queries with known-correct answers before you tune anything, because without that baseline, every change to chunking, retrieval, or prompting is a guess you cannot defend when someone asks whether the system got better or worse.
Retrieval Metrics: Recall@k and MRR
Recall@k measures whether the chunk containing the correct answer appears anywhere in the top k retrieved results, and it is the single most diagnostic metric in a RAG system because if the right chunk was never retrieved, no amount of prompt engineering downstream can fix the answer. Mean reciprocal rank goes a step further and rewards ranking the correct chunk higher: it averages 1 divided by the rank of the first correct result across all queries, so a system that consistently ranks the right chunk first scores near 1.0 while one that buries it at rank 8 scores much lower even with perfect recall@10. Precision@k, the fraction of retrieved chunks that are actually relevant, matters too, since a generator fed mostly irrelevant context tends to produce vaguer, less faithful answers even when the one correct chunk is technically present.
- Recall@k: did the correct chunk appear anywhere in the top k results
- MRR: how highly was the first correct chunk ranked, averaged across all queries
- Precision@k: what fraction of retrieved chunks were actually relevant to the query
Generation Metrics: Faithfulness and Answer Relevance
Faithfulness measures whether the generated answer is actually supported by the retrieved context, catching the specific failure where a model answers correctly by coincidence or from its own training data rather than from what was retrieved, which matters because that behavior is exactly what breaks when the underlying facts in your documents change. Answer relevance measures whether the response actually addresses the question asked, independent of whether it is factually grounded, since a perfectly faithful answer to a slightly different question than the one asked is still a failure from the user's perspective. Most production teams score both with an LLM-as-judge approach, similar to the RAGAS framework, prompting a separate model to score faithfulness and relevance against the retrieved context and the original query, though judge reliability itself needs periodic spot-checking against human review.
Building a Golden Evaluation Set That Doesn't Lie to You
Pull 150 to 300 real queries from actual usage logs or stakeholder interviews rather than inventing plausible-sounding questions, and pair each with the chunk or chunks that should be retrieved and, where possible, a reference answer. Stratify deliberately: include queries with exact identifiers, ambiguous phrasing, multi-part questions, and queries where the honest answer is that the corpus does not contain the information. Freeze and version the set in source control so results are comparable across changes, and hold back a slice the team building the system never sees, reserved for final acceptance testing so the eval set was never implicitly optimized against.
Where RAG Systems Actually Break: A Diagnostic Framework
Low recall@k with otherwise reasonable performance elsewhere points to a chunking or embedding problem: either the right information is not being retrieved at all, or it is split across chunks awkwardly. High recall@k paired with low faithfulness usually points to a generation problem: the right context was retrieved but the prompt is not constraining the model to use it, or too much irrelevant context is diluting the signal. High faithfulness paired with low answer relevance usually means the retrieved context is technically correct but incomplete for the actual question, which often means precision@k is too low and a reranker is needed. Running retrieval and generation metrics separately is what makes this diagnosis possible; a single blended score just tells you something is wrong without telling you what.
How Netray Builds RAG Evaluation Harnesses
Every RAG system Netray ships includes a versioned evaluation harness the client owns and can rerun independently, built from a golden set assembled from the client's own real query history rather than generic benchmark questions. Our DataRay platform includes a dashboard reporting recall@k, MRR, faithfulness, and answer relevance continuously, with automated regression checks on every chunking, retrieval, or prompt change so a regression surfaces before users notice rather than after. For clients running on-premises, the entire evaluation pipeline runs inside their boundary using the same on-prem models serving production traffic.
Frequently Asked Questions
What is recall@k in RAG evaluation?
Recall@k measures whether the chunk containing the correct answer to a query appears anywhere within the top k results returned by the retriever. It is the most diagnostic retrieval metric because it isolates whether the system even found the right information, independent of how the generation model later used it. A common practice is to report recall at several values of k, such as 5, 10, and 20, to see how quickly correct chunks fall out of the retrieved set.
What does faithfulness mean in a RAG evaluation?
Faithfulness measures whether a generated answer is actually supported by the content that was retrieved, rather than drawn from the model's own training data or invented outright. It catches a specific and important failure mode: an answer that happens to be correct despite not being grounded in the retrieved context will break the moment the underlying facts change, since the model was never actually using the source material to begin with.
Can LLMs reliably judge RAG answer quality?
LLM-as-judge approaches, similar to the RAGAS framework, are useful and widely used for scoring faithfulness and relevance at scale, but they are not perfectly reliable and need periodic validation against human review, particularly on ambiguous or borderline cases. Treat LLM-judged scores as a strong signal for tracking trends and catching regressions, and reserve human review for final acceptance decisions and any case where the judge score itself looks surprising.
How many queries should a RAG golden evaluation set contain?
Between 150 and 300 stratified real queries is enough for most enterprise RAG systems to produce statistically meaningful comparisons between architecture changes. Below roughly 100 queries, the confidence interval on any metric gets wide enough that a five-point change is indistinguishable from noise. Stratify deliberately to include exact-match queries, ambiguous phrasing, and queries where the honest answer is that the corpus lacks the information.
Key Takeaways
- 1Retrieval Metrics: Recall@k and MRR: Recall@k measures whether the chunk containing the correct answer appears anywhere in the top k retrieved results, and it is the single most diagnostic metric in a RAG system because if the right chunk was never retrieved, no amount of prompt engineering downstream can fix the answer. Mean reciprocal rank goes a step further and rewards ranking the correct chunk higher: it averages 1 divided by the rank of the first correct result across all queries, so a system that consistently ranks the right chunk first scores near 1.0 while one that buries it at rank 8 scores much lower even with perfect recall@10.
- 2Generation Metrics: Faithfulness and Answer Relevance: Faithfulness measures whether the generated answer is actually supported by the retrieved context, catching the specific failure where a model answers correctly by coincidence or from its own training data rather than from what was retrieved, which matters because that behavior is exactly what breaks when the underlying facts in your documents change. Answer relevance measures whether the response actually addresses the question asked, independent of whether it is factually grounded, since a perfectly faithful answer to a slightly different question than the one asked is still a failure from the user's perspective.
- 3Building a Golden Evaluation Set That Doesn't Lie to You: Pull 150 to 300 real queries from actual usage logs or stakeholder interviews rather than inventing plausible-sounding questions, and pair each with the chunk or chunks that should be retrieved and, where possible, a reference answer. Stratify deliberately: include queries with exact identifiers, ambiguous phrasing, multi-part questions, and queries where the honest answer is that the corpus does not contain the information.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
RAG Accuracy Readiness Assessment
Score your retrieval-augmented generation system across eight dimensions that actually predict production accuracy, from chunking strategy to groundedness verification.
Free ToolAI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
Free ToolLLM Evaluation Readiness Assessment
Score your organization across ten dimensions of LLM evaluation maturity, from golden datasets and regression gates to human review and production monitoring.
Terms used in this article
Shipping a RAG system without a real evaluation harness? Netray will build the golden set and metrics dashboard from your own query history before you tune another prompt.
Related Resources
Enterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
AI & AutomationRAG Chunking Strategies: Fixed, Semantic, Structural, and Late
Compare RAG chunking strategies, fixed-size, semantic, structural, and late chunking, with concrete guidance on chunk size, overlap, and when each wins.
AI & AutomationHybrid Search: Combining BM25 and Vector Retrieval with RRF
Hybrid search for RAG: why pure vector retrieval misses exact matches, how BM25 fixes it, and how reciprocal rank fusion combines both reliably.