Retrieval-Augmented Generation (RAG) grounds a language model in an external corpus so answers cite real documents instead of hallucinating from parametric memory. The pattern is deceptively simple — retrieve relevant chunks, stuff them into the prompt, generate — but the interesting engineering lives in every seam between those steps.
The pipeline in practice
A production RAG system is a chain of failure points. Chunking strategy decides what a "unit of meaning" is. Embedding choice decides what "similar" means. Retrieval decides what the model even sees. If any stage drops the relevant passage, no amount of prompt cleverness recovers it — garbage retrieval is garbage generation.
Most of the wins I have shipped came from the retrieval half, not the model. Hybrid search (dense vectors plus BM25), reranking the top-k, and query rewriting consistently beat swapping in a bigger model. That is why the storage layer matters so much: see [[Vector Databases]] for how the index shapes recall.
Where it goes wrong
The classic trap is treating RAG as a one-shot lookup. Real questions are multi-hop, ambiguous, or need decomposition. Good [[Prompt Engineering]] turns a vague user query into a retrieval-friendly one and instructs the model to abstain when context is thin — abstention is a feature, not a bug.
You cannot improve what you cannot see. Logging retrieved chunks, their scores, and whether the answer actually used them is the backbone of [[LLM Observability]]. Without that trace, every regression is a guessing game.
The honest summary
RAG is not a model technique; it is a search system with a language model bolted on the end. Treat retrieval quality as the primary metric, instrument the whole chain, and the generation mostly takes care of itself. When agents are involved, the same grounding discipline feeds directly into [[Agent Evaluation]].