Technical Report: Modern RAG Architectures

TLDR

The Retrieval-Augmented Generation (RAG) landscape has undergone a paradigm shift from linear "Retrieve-and-Read" pipelines to complex Modular RAG architectures. As of 2025, production-grade systems prioritize Hybrid Retrieval—combining dense semantic vectors with sparse BM25 keyword search—and Post-Retrieval Reranking to mitigate the "Lost in the Middle" phenomenon. Advanced implementations now leverage Contextual Retrieval to preserve document-level semantics and GraphRAG for global reasoning across massive datasets. Evaluation has matured through the RAGAS framework, focusing on the "RAG Triad": Faithfulness, Answer Relevance, and Context Relevance. This report details the technical transition from naive patterns to agentic, multi-stage systems, emphasizing the necessity of A (comparing prompt variants) to optimize performance.

---

Conceptual Overview

The fundamental goal of RAG is to bridge the gap between a Large Language Model's (LLM) static parametric knowledge and dynamic, private, or real-time data. By decoupling knowledge from the model's weights and placing it in an external retrieval corpus, developers can mitigate hallucinations, provide verifiable citations, and update the system's knowledge base without expensive retraining.

The Evolution: From Naive to Modular

The architectural journey of RAG is categorized into three distinct generations:

1. Naive RAG: A simple "Retrieve-Read" pattern. The system takes a user query, converts it into a vector, finds the top-k similar chunks in a vector database, and stuffs them into the LLM prompt. This approach often fails due to low precision (retrieving irrelevant chunks) and low recall (missing relevant information due to poor embedding alignment).
2. Advanced RAG: Introduced pre-retrieval and post-retrieval optimizations. Techniques like Query Expansion (generating multiple versions of a query) and Reranking (using a cross-encoder to re-evaluate the top-k results) significantly improved performance.
3. Modular RAG: The current state-of-the-art. It breaks the pipeline into interchangeable modules: query transformation, routing, indexing, retrieval, and refinement. This allows for specialized workflows, such as routing a query to a SQL database for structured data or a vector store for unstructured text.

Addressing Semantic Dilution and "Lost in the Middle"

A critical challenge in modern architectures is the "Lost in the Middle" phenomenon, identified by Liu et al. (2023). Research shows that LLMs demonstrate high performance when relevant information is at the very beginning or end of the context window but struggle when it is buried in the center.

Furthermore, "semantic dilution" occurs when arbitrary chunking (e.g., every 512 tokens) severs the relationship between a sentence and its broader document context. Modern RAG architectures address this through Contextual Retrieval and Prompt Compression, ensuring that the most salient information is positioned optimally for the model's attention mechanism and that every chunk carries its document-level "DNA."

', 'Knowledge Graph', and 'RAGAS Evaluator' loop. Arrows indicate bi-directional data flow in the Modular stage.)

---

Practical Implementations

Implementing a production-grade RAG system in 2025 requires a sophisticated stack and a focus on hybrid methodologies.

The Modern Tech Stack

The ecosystem has converged on a few key components:

• Orchestration: LangChain and LlamaIndex remain the dominant frameworks for building the "plumbing" of RAG. They provide standardized interfaces for data connectors, chunking strategies, and agentic loops.
• Vector Databases: Specialized engines like Pinecone, Qdrant, and Weaviate handle high-dimensional similarity searches. These databases now support HNSW (Hierarchical Navigable Small World) graphs for low-latency retrieval at scale.
• Embedding Models: Models like OpenAI’s text-embedding-3-large or open-source alternatives like BGE-M3 provide the semantic foundation.

Hybrid Retrieval: BM25 + Dense Embeddings

Relying solely on vector embeddings (dense retrieval) often leads to failures in "out-of-vocabulary" scenarios or when searching for specific serial numbers or technical terms. Modern systems use Hybrid Retrieval:

1. Dense Search: Captures semantic meaning (e.g., "feline" matches "cat").
2. Sparse Search (BM25): Captures exact keyword matches (e.g., "Model-X-123").
3. Reciprocal Rank Fusion (RRF): A mathematical algorithm used to combine the results from both searches into a single, ranked list. The formula: $$RRFscore(d) = \sum_{r \in R} \frac{1}{k + r(d)}$$ (where $k$ is a constant, usually 60) ensures that documents appearing high in both lists are prioritized.

Contextual Retrieval and Chunking

Standard chunking often breaks semantic continuity. Contextual Retrieval, popularized by Anthropic, involves prepending a document-level summary to every chunk before indexing.

• Example: Instead of a chunk saying "The revenue increased by 5%," the contextualized chunk says "[Summary: This is the 2024 Q3 Financial Report for TechCorp] The revenue increased by 5%." This ensures that the embedding captures the "global" context of the chunk, drastically reducing retrieval errors.

Post-Retrieval Reranking

Retrieval is often a trade-off between speed and accuracy. To optimize this, engineers use a two-stage process:

1. Retrieval: Use a fast bi-encoder (vector search) to get the top 100 candidates.
2. Reranking: Use a computationally expensive Cross-Encoder (like Cohere Rerank or BGE-Reranker) to score the relationship between the query and each of the 100 candidates, selecting the top 5-10 for the final prompt. This effectively solves the "Lost in the Middle" problem by ensuring the most relevant data is at the top.

---

Advanced Techniques

As systems move beyond simple retrieval, two paradigms have emerged: Agentic RAG and GraphRAG.

Agentic RAG: Autonomous Iteration

In Agentic RAG, the LLM is not just a passive reader but an active controller. The agent follows a ReAct (Reason + Act) pattern:

1. Analyze: Does the retrieved context answer the user's question?
2. Refine: If not, what is missing? The agent might generate a new search query or look into a different data source (e.g., a web search or a SQL database).
3. Validate: Once an answer is generated, the agent checks it against the source context to ensure no hallucinations occurred. This iterative loop allows the system to handle multi-step questions like "Compare the revenue growth of Company A and Company B over the last three years," which requires multiple distinct retrieval steps.

GraphRAG: Reasoning Across Knowledge Domains

While vector search is excellent at finding "local" similarity (specific facts), it struggles with "global" reasoning (summarizing themes across a whole dataset). GraphRAG (Microsoft Research, 2024) solves this by:

1. Entity Extraction: Identifying all people, places, and concepts in the corpus.
2. Relationship Mapping: Building a Knowledge Graph where nodes are entities and edges are their relationships.
3. Community Detection: Using algorithms like Leiden to group related entities into "communities."
4. Hierarchical Summarization: Generating summaries for each community. When a user asks a global question ("What are the main themes in these 1,000 legal documents?"), GraphRAG retrieves the community summaries rather than individual text chunks, providing a comprehensive overview that vector search would miss.

 in a 3D space. Right side: GraphRAG showing a query hitting a 'web' of interconnected nodes, highlighting a cluster of related entities and their pre-generated summary.)

---

Research and Future Directions

The frontier of RAG research is currently focused on automated evaluation and the convergence of retrieval with native model capabilities.

The RAGAS Framework and the RAG Triad

Traditional metrics like BLEU or ROUGE are insufficient for RAG because they only measure text overlap, not factual accuracy. The RAGAS framework uses an "LLM-as-a-judge" to quantify the RAG Triad:

• Faithfulness: Is the answer derived only from the context? (Prevents hallucinations).
• Answer Relevance: Does the answer actually address the user's prompt?
• Context Relevance: Was the retrieved context necessary and sufficient to answer the question?

Optimization via A Testing

Engineers use A (comparing prompt variants) to test different system prompts and retrieval depths. By running the same query set through Pipeline A (e.g., 512-token chunks) and Pipeline B (e.g., 256-token chunks with contextual summaries), and comparing their RAGAS scores, teams can scientifically determine the optimal configuration for their specific domain.

Long-Context RAG and Native Retrieval

With the advent of models supporting 1M+ token context windows (like Gemini 1.5 Pro), some argue that RAG is obsolete. However, research suggests that:

1. Cost Efficiency: Retrieving 5 relevant chunks is significantly cheaper than feeding 1 million tokens into every prompt.
2. Performance: Even long-context models suffer from performance degradation as the context grows. The future likely holds a hybrid approach where RAG acts as a "filter" to select the most relevant 50k–100k tokens, which are then processed by a long-context model.

---

Frequently Asked Questions

Q: How do I choose between a Vector Database and a Knowledge Graph for RAG?

Vector databases are best for "needle-in-a-haystack" queries where you need to find specific facts. Knowledge Graphs (GraphRAG) are superior for "global" queries that require understanding relationships or summarizing themes across the entire dataset. Most modern enterprise systems are moving toward a hybrid "Graph-Vector" approach.

Q: What is the most effective way to prevent hallucinations in RAG?

The most effective method is a combination of Post-Retrieval Reranking (to ensure only the most relevant context reaches the LLM) and Faithfulness Evaluation using RAGAS. Additionally, implementing a "Chain of Verification" prompt, where the LLM must cite specific chunk IDs for every claim, significantly reduces grounding errors.

Q: Does chunk size matter in 2025?

Yes. While models have larger windows, smaller chunks (256–512 tokens) usually provide higher retrieval precision. However, the "Contextual Retrieval" technique (adding summaries to chunks) allows you to use smaller chunks without losing the broader document context, offering the best of both worlds.

Q: What is the "Lost in the Middle" problem?

It is a documented behavior where LLMs are better at recalling information at the start and end of a long prompt. In RAG, if your most relevant chunk is the 5th out of 10 retrieved chunks, the model might ignore it. Solving this requires Reranking to move that 5th chunk to the 1st position.

Q: How does "A" testing apply to RAG?

In the context of RAG, A (comparing prompt variants) involves running the same set of queries through two different pipeline configurations—such as different chunking sizes, different embedding models, or different system prompts—and comparing their RAGAS scores to determine which configuration yields higher faithfulness and relevance.