Definition
A data structure mapping unique tokens or keywords to their specific locations within document chunks, enabling high-speed lexical retrieval in RAG pipelines; it offers superior precision for exact-match queries (like serial numbers) but fails to capture semantic synonyms compared to vector indexes.
Lexical keyword lookup (BM25) vs. Semantic similarity lookup (Vector Search).
"The alphabetical index at the back of a technical textbook that lists every page number where a specific term appears."
- BM25(Prerequisite Ranking Algorithm)
- Hybrid Search(Implementation Pattern)
- Vector Index(Complementary Component)
- Sparse Embeddings(Modern Alternative Representation)
Conceptual Overview
A data structure mapping unique tokens or keywords to their specific locations within document chunks, enabling high-speed lexical retrieval in RAG pipelines; it offers superior precision for exact-match queries (like serial numbers) but fails to capture semantic synonyms compared to vector indexes.
Disambiguation
Lexical keyword lookup (BM25) vs. Semantic similarity lookup (Vector Search).
Visual Analog
The alphabetical index at the back of a technical textbook that lists every page number where a specific term appears.