RAG: understanding the path from retrieval to response
We followed RAG retrieval results through the text sent to the model and the evaluated answer. We want to identify which stage to change and how to measure its effects.
We studied how a RAG system prepares answers to understand where information might be dropped and how to observe that loss. We want reference points for deciding what to improve in our solutions: passage retrieval, passage selection or answer evaluation.
A RAG system begins by searching documents for passages. The application then prepares the text it will send to the model: it selects passages, orders them and may add source titles or identifiers. The model receives this text with the question, then produces an answer.
We focused on that preparation stage. A retriever may have found a useful sentence without the application transmitting it to the model. To check this, we preserved retrieval results, the actual transmitted context and the generated answer separately. We can therefore examine information’s path instead of directly attributing failure to the retriever or model.
What we followed in an initial case
We identified an example in our exploratory cohort, separate from the 256 questions in the primary comparison. The question concerns the company chaired by Luca Cordero di Montezemolo that merged into Fiat Chrysler Automobiles. The dataset annotates three supporting sentences: they connect the person to the company and describe the company and its merger.
With BM25, we retrieve all three sentences within the top ten passages. We then build the context with a 512-token budget. Two of the three annotated sentences are included; the passage connecting Montezemolo to the company does not fit. In this execution, the model abstains.
At 1,536 tokens, we transmit all three sentences along with other passages. The model then answers “Fiat S.p.A.” and cites the sentences connecting the person to the company and describing the merger. We preserved the question, contexts and answers in the archived case.
We use this example to make the change between retrieval and the model call visible. It does not explain every abstention or measure a general effect. We therefore organized a larger question comparison, keeping retrieval results unchanged between budgets.
How we organized the comparison
We use HotpotQA, introduced by Yang and colleagues in 2018, because its questions include sentences annotated as supporting facts. These give us reference points to locate in retrieved passages and transmitted context. They are not an exhaustive account of all possible evidence.
We selected 256 questions, half bridge questions connecting information and half comparison questions. For each, we search the ten paragraphs supplied by the dataset. We do not perform an open search across Wikipedia.
We ranked three-sentence chunks using BM25, which relies on words, and dense retrieval, which uses numerical text representations. We retained the top ten passages from each ranking and compared two selection rules: follow the ranking or alternate between documents. We used neither support annotations nor reference answers to guide those choices.
We compared budgets of 512 and 1,536 tokens. A token is a unit of text, often a word or part of a word. We count the whole documentary input, including titles and identifiers. When a passage does not fit in the remaining space, we skip it in full.
Our first measure counts questions whose resulting context still contains every annotated sentence. We are following text preservation here. We then examine separately what that measure lets us say about answers.
What we observed when changing the context
Annotation retention by budget and selection rule
We examined annotated-sentence retention and lexical answer agreement separately. They concern different question populations; use the selector to move between the two measures.
Population: all 256 questions in the preservation study.
Word matching (BM25)
Mean share of annotated passages retrieved in the top ten, across 256 questions: 94.8%.
Dense retrieval (embeddings)
Mean share of annotated passages retrieved in the top ten, across 256 questions: 98.9%.
Questions, out of 256
All annotated sentences retained: BM25 148 to 228 /256; dense retrieval 220 to 249 /256.
We checked whether every annotated sentence was present in the transmitted context. This does not establish that the evidence is sufficient. We tested document alternation; it is not MMR.
| Every annotated sentence retained, out of 256 questions | 512 tokens | 1,536 tokens |
|---|---|---|
| BM25, following the search ranking | 148 | 228 |
| Dense retrieval, same packing rule | 220 | 249 |
With BM25, we move from 148 to 228 questions with every annotation transmitted. We did not change retrieval between budgets. We therefore observe a difference introduced during context assembly: 80 additional questions retain all their annotated sentences when we allow more space.
We calculated a gain of 31.25 percentage points, with a simultaneous interval of +24.2 to +38.7 within the comparison family. Dense retrieval moves from 220 to 249 questions, approximately +11.3 points, with an interval of +6.6 to +16.4. We preserve paired calculations and uncertainty in the repository.
We also wanted to check whether alternating documents helps prevent one source occupying most of the context. At 512 tokens with dense retrieval, this rule preserves all annotations for 203 questions, against 220 when following the ranking. The approximately -6.6-point difference remains established after Holm correction; the simultaneous interval runs from -11.3 to -2.3.
We identify a limitation of this simple rule: making room for more documents can also displace complementary passages from the same document. We therefore need to evaluate what selection preserves rather than treating source variety as sufficient benefit in itself.
How we relate the results to existing methods
Selecting a document set has long been studied. In 1998, Carbonell and Goldstein proposed MMR, combining relevance and novelty relative to selected results. In SetR at ACL 2025, Lee and colleagues guide selection through the information needed to answer a question.
These studies explain why we should look beyond individual rankings. Our measurements describe what happens in our apparatus when we fit passages into a budget. We tested document alternation, which is neither MMR nor SetR. We cannot compare those methods’ efficacy from our numbers.
We also want to examine compression. RECOMP at ICLR 2024 studies sentence selection and summaries of multiple documents. LongLLMLingua at ACL 2024 combines question-guided compression with document reordering. These approaches may help us prepare shorter contexts, but we need to compare them under their own conditions of use.
We will also need to adapt evaluation: a summary can preserve a sentence’s meaning without preserving its words. Counting original sentences would no longer suffice to establish whether information disappeared. We would need judgements of meaning and documentary support.
Finally, Liu and colleagues’ Lost in the Middle study in 2024 shows that information position can influence its use in certain long contexts. We did not vary position independently of other factors. We retain that question for a dedicated comparison, without attributing our campaign’s results to it.
What we can say about generated answers
For the main generation study, we selected 128 of the 256 questions and used eight configurations, producing 1,024 outputs. We called Qwen3.6 36B in Q4_K_M quantization through Ollama and used nomic-embed-text for dense retrieval. We preserve digests and parameters to identify these models precisely.
With ranking-based selection, we count 60 exact reference matches at 512 tokens and 80 at 1,536 for BM25. For dense retrieval, we count 79 and 86. The BM25 change is established for this lexical endpoint; the dense change remains inconclusive within the test family.
We cannot interpret those counts as the number of factually correct answers. In the Fiat case above, the reference expects “Fabbrica Italiana Automobili Torino”. The answer “Fiat S.p.A.” receives zero Exact Match even though the supplied paragraph identifies the names as referring to the same company. We preserve the official score, but need to examine the texts to understand what it measures.
In the primary campaign diagnostics, we also identify 39 of 128 questions combining every annotation present with zero Exact Match for BM25 at 1,536 tokens. Three combine incomplete annotated context with an exact match. We still need to distinguish differences in wording, abstentions, prior knowledge and annotation limitations.
We draw on the distinctions proposed by RAGAS at EACL 2024 and the combination of automated judges and human annotations in ARES at NAACL 2024 to prepare this next stage. We organized an independent dual-annotation workflow. We still need the 1,024 outputs annotated separately for semantic correctness, documentary support, citation validity and justified abstention.
What we want to reuse in development
For a given answer, we want to explain which passages were retrieved, which were transmitted and why some were dropped. We can then choose a change and check its effects on the same questions. This is the function we envisage for an assembly component shared across our solutions.
We want to bring together actual-budget counting, references to source sentences and selection traces. We could then integrate existing methods and compare them at equal budgets. We will need to do this without HotpotQA’s annotations, which are generally unavailable in enterprise documentation.
In our study repository, we preserved texts, answers, vectors and programs for reproducing deterministic calculations without a model server. We also performed exploratory retrieval across a pooled corpus of 2,547 paragraphs, without generation. This remains different from an organization’s documentation; we still need to check the apparatus in that setting.
We therefore have a foundation for two further tasks: completing answer judgements, then comparing assembly choices. We can decide on component reuse and publication from those evaluations. We are extending our LLM Architecture & RAG Pipelines work, presented on the AI services page.