A pragmatist's guide to RAG (with code)

RAG is not a silver bullet. Here's when it works, when it doesn't, and how to know which problem you're actually solving.

The hype around RAG is that it lets you ground LLMs in fresh data. In practice, RAG is a retrieval problem disguised as an AI problem. Most failures are at the retrieval stage, not the generation stage.

When RAG wins

You have a corpus of documents and need the LLM to cite them accurately. RAG shines here. The retrieval step finds relevant context; the LLM synthesizes. You get citations for free.

When fine-tuning wins

You need the model to learn a new way of reasoning or style of output. Fine-tuning is the right tool. RAG won't teach the model anything; it only provides context.

The hard part is knowing which bucket your problem fits into. Hint: if you're uncertain, start with RAG. It's easier to debug.