LLMs need access to knowledge beyond their training data. The two primary approaches are RAG (retrieve relevant chunks at query time) and long context (stuff the full knowledge base into the context window). As context windows have grown from 4K tokens to the 1M+ token windows now common across current frontier models from Anthropic, OpenAI, and Google, the tradeoffs between these approaches have shifted. The simple “bigger window kills RAG” framing has not held up: as of 2026, the two are most often combined rather than chosen between.

Overview

AspectRAGLong Context
Knowledge VolumeUnlimited (external store)Limited by context window
Retrieval QualityDepends on retrieval pipelineAll information available
LatencyRetrieval adds latencyHigher first-token latency
Cost Per QueryLower (smaller prompts)Higher (large context)
FreshnessReal-time (if index is current)Requires re-constructing context
AccuracyCan miss relevant chunksCan lose focus in large contexts
InfrastructureVector DB + embeddings + chunkingNone beyond the LLM

How RAG Works

RAG retrieves relevant document chunks based on the user’s query, then includes those chunks in the LLM’s context. The pipeline involves document chunking, embedding generation, vector storage, retrieval (often hybrid keyword + vector), and prompt construction. The model sees only the retrieved chunks, not the full knowledge base.

How Long Context Works

Long context approaches load the entire relevant knowledge base into the prompt. With context windows reaching 1M+ tokens, you can include hundreds of pages of documentation, codebases, or conversation history. The model has access to everything and can find relevant information through its attention mechanism.

Retrieval Quality

RAG’s retrieval step is both its strength and weakness. Good retrieval surfaces the most relevant information efficiently. Bad retrieval misses critical chunks, leading to incomplete or incorrect answers. Retrieval quality depends on chunking strategy, embedding model quality, query formulation, and hybrid search configuration. Building a high-quality retrieval pipeline requires significant engineering effort.

Long context avoids retrieval errors entirely - everything is in the context. However, advertised context windows do not equal usable context. Models can exhibit “lost in the middle” behavior where information in the center of very long contexts gets less attention than information at the beginning or end. More broadly, “context rot” describes how reliability tends to decline as input length grows, even well below the stated window limit and even on simple tasks. Chroma’s 2025 study tested 18 production models (including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3) and found that performance degraded as input length increased across all of them, with the effect worsening when distractors are present or when the relevant passage is less semantically similar to the query. The takeaway is that filling a 1M token window is not the same as the model reliably using all of it.

Cost and Latency

RAG is more cost-effective per query. A typical RAG prompt includes 1-5K tokens of retrieved context. Long context prompts can be 100K-1M+ tokens. At current per-token pricing, the cost difference is orders of magnitude for large knowledge bases.

Prompt caching narrows this gap when the same large context is reused across many requests. Anthropic prices cache reads at 0.1x the base input price (a 90% discount) with a 5-minute default cache lifetime (a 1-hour option is available), Google Gemini offers context caching billed by stored token-hours, and OpenAI applies automatic caching to repeated prompt prefixes. Caching only helps when the bulk of the context is stable between calls, so it favors long context over a fixed corpus, not RAG where the retrieved chunks change every query.

Latency profiles differ. RAG adds retrieval latency (typically tens to low hundreds of milliseconds for vector search) but has lower model latency due to shorter prompts. Long context eliminates retrieval latency but increases time-to-first-token roughly in proportion to context length, unless the prefix is served from cache. For very large uncached contexts, the model processing time can be substantial.

Freshness and Updates

RAG supports real-time knowledge updates. When documents change, you re-index them, and subsequent queries retrieve the updated content. This makes RAG ideal for rapidly changing knowledge bases.

Long context requires reconstructing the prompt when knowledge changes. For static or slowly changing knowledge bases, this is manageable. For frequently updated content, long context is less practical.

When to Choose RAG

Choose RAG when your knowledge base exceeds the practical context window size, when cost per query matters, when knowledge updates frequently, when you need to cite specific sources in responses, or when your knowledge base spans diverse topics where only a small fraction is relevant to any given query.

When to Choose Long Context

Choose long context when your knowledge base fits within the context window, when retrieval quality is a concern, when the full context is needed for reasoning (code analysis, document comparison), or when the engineering cost of building and maintaining a RAG pipeline is not justified for your use case.

Practical Recommendation

RAG and long context are not mutually exclusive, and in 2026 the dominant pattern is to combine them. Use RAG to narrow a large corpus down to the most relevant material, then use a long context window to hold those retrieved chunks plus conversation history and tool outputs so the model can reason over sufficient context. This is also where “agentic RAG” sits: instead of a single retrieve-then-answer pass, an agent plans queries, retrieves in multiple hops, re-ranks, and decides what to keep, which is closer to context engineering than to a static pipeline. For small knowledge bases (under 100 pages or so), try long context first - it is simpler and avoids retrieval errors. For larger knowledge bases, RAG remains essential for both cost and quality, and retrieval also gives you something long context alone does not: citable, attributable sources. If your decision is really retrieval versus changing the model’s weights, see RAG vs fine-tuning ; for managing what lands in the window once you have the context, see context engineering vs prompt engineering .

Sources