Beyond Relevance: Utility-Centric Retrieval in the LLM Era
Retrieval systems built for search engines may be wrong for AI — this paper argues 'useful to an LLM' should replace 'topically relevant.'
The Thesis
Most retrieval systems today are optimized to find documents that are topically relevant to a query — matching keywords and subject matter. But when an LLM is the consumer of those documents rather than a human reader, topical relevance is the wrong target. What matters is whether the retrieved text actually helps the model produce a better answer — a property the authors call 'utility.' This paper proposes a conceptual framework for redesigning retrieval pipelines around LLM utility, covering cases where the retrieval system is aware of the model it feeds versus cases where it is not. The catch: this is a tutorial paper, not an empirical study, so the framework is theoretical and the hard engineering work remains ahead.
Catalyst
Retrieval-augmented generation (RAG) — the practice of feeding live documents to an LLM before it answers — has gone from research novelty to production standard in roughly two years. As RAG deployments scale, practitioners are discovering that high relevance scores from classic search tools don't reliably improve LLM output quality. That gap between retrieval metrics and downstream generation quality is the practical problem this paper tries to formalize.
What's New
Classical information retrieval, as practiced in systems like BM25 (a keyword-matching algorithm) or early dense retrievers like DPR (Dense Passage Retrieval), optimizes for ranking documents by topical match to a query. Those systems were designed for humans who would read the results directly. This paper argues that when an LLM is the downstream consumer, a different objective — whether the document improves the model's answer — should govern retrieval design, and it synthesizes recent work toward a unified framework for that shift.
The Counter
This is a tutorial paper, not an experimental one — it proposes a framework and synthesizes existing literature but does not run a single benchmark comparing relevance-centric versus utility-centric retrieval on real tasks. The central claim that 'utility' should replace 'relevance' is intuitive but underspecified: utility for which LLM, on which task, measured how? In practice, LLM output quality is expensive and noisy to evaluate, making it a poor training signal for retrieval systems that need to rank millions of documents in milliseconds. There is also a chicken-and-egg problem: utility-aware retrieval requires tight coupling to a specific LLM, but production systems often swap models, update prompts, or serve multiple models simultaneously. Finally, recent empirical work has shown that with good chunking and embedding strategies, relevance-tuned retrievers already transfer well to RAG settings — suggesting the gap this paper frames as fundamental may be narrower than claimed.
Longs
- ESTC (Elastic) — core search infrastructure underpinning many RAG deployments
- MDB (MongoDB) — vector search capabilities built into document databases
- MSFT — Azure AI Search and Copilot RAG stack
- AIXI / BOTZ (AI infrastructure ETFs) — broad exposure to enterprise AI tooling
Shorts
- Pure-play keyword search vendors (legacy FAST/Solr deployments) — their relevance-tuning moat shrinks if utility metrics displace BM25-style ranking
- Enterprise search incumbents like Coveo or Sinequa — their scoring models are relevance-centric and would need rebuilding for LLM-utility optimization
Enablers (Picks & Shovels)
- LlamaIndex — open-source RAG orchestration framework
- LangChain — retrieval pipeline tooling widely used in production
- Hugging Face MTEB benchmark — the standard for measuring embedding/retrieval quality
- BEIR benchmark — heterogeneous retrieval evaluation suite used to test generalization
Private Watchlist
- Vectara — RAG-focused retrieval platform
- Cohere — enterprise LLM with retrieval integration
- Unstructured — document preprocessing for RAG pipelines
- Contextual AI — RAG-native enterprise model provider
Resources
The Paper
Information retrieval systems have traditionally optimized for topical relevance-the degree to which retrieved documents match a query. However, relevance only approximates a deeper goal: utility, namely, whether retrieved information helps accomplish a user's underlying task. The emergence of retrieval-augmented generation (RAG) fundamentally changes this paradigm. Retrieved documents are no longer consumed directly by users but instead serve as evidence for large language models (LLMs) that produce answers. As a result, retrieval effectiveness must be evaluated by its contribution to generation quality rather than by relevance-based ranking metrics alone. This tutorial argues that retrieval objectives are evolving from relevance-centric optimization toward LLM-centric utility. We present a unified framework covering LLM-agnostic versus LLM-specific utility, context-independent versus context-dependent utility, and the connection with LLM information needs and agentic RAG. By synthesizing recent advances, the tutorial provides conceptual foundations and practical guidance for designing retrieval systems aligned with the requirements of LLM-based information access.