RAG Explained With Real Code (And When You Shouldn't Use It)

A language model knows what it was trained on and nothing else. It has never seen your company's documents, last week's support tickets, or the policy you published yesterday. Ask it about them and it will either admit ignorance or, worse, invent a confident answer. Retrieval-augmented generation — RAG — is the standard fix, and it is genuinely elegant. It is also reached for far too often. Let us do both halves: how it works, and when you should not bother.

The problem RAG solves

The naive approach to "make the model know my data" is to paste all your data into the prompt. That fails immediately — there is a limit to how much you can send, it is expensive, and a model buried in a thousand pages answers worse, not better. The opposite approach, retraining the model on your data, is slow, costly, and has to be redone every time the data changes.

RAG threads the needle. Instead of sending everything, you send only the handful of passages relevant to the current question, fetched on demand. The model gets exactly the context it needs, nothing it does not, and your data can change every minute without retraining anything.

How it works, step by step

RAG has two phases. Once, up front, you prepare your data. Then, on every question, you retrieve and generate.

The clever part is the middle. Every chunk of your text is converted into a vector — a list of numbers that captures its meaning — and stored. The user's question is converted the same way, and you find the chunks whose vectors are closest to the question's. "Closest" here means closest in meaning, not in wording, which is why RAG can find the right passage even when the user's phrasing shares no words with it.

The whole thing in code

Stripped to its essentials, a RAG system is short. This is the entire idea in a handful of lines.

RAG, minus the frameworks# ONE-TIME: prepare the knowledge base
chunks = split_documents(docs, size=500)      # small, focused pieces
vectors = embed(chunks)                     # meaning to numbers
store.add(chunks, vectors)                  # into a vector database

# PER QUESTION: retrieve, then generate
def answer(question):
    q_vec = embed([question])
    top = store.search(q_vec, k=3)         # 3 closest chunks
    context = "\n".join(top)
    return llm(f"Answer using only:\n{context}\n\nQ: {question}")

That is the whole architecture. Everything else — the frameworks, the vector databases, the reranking models — is refinement on top of these two phases. Understanding this skeleton is what lets you debug a RAG system when it misbehaves, because you know the three places it can go wrong: bad chunking, bad retrieval, or bad generation.

When not to use it

Here is the part the tutorials skip. RAG is machinery, and machinery has a cost — a vector database to run, an embedding step on every query, more things to break. Plenty of problems do not need it.

If your knowledge base is small and static — a few pages that rarely change — just put it in the system prompt. It is simpler, faster, and often cheaper than standing up a retrieval pipeline. If the model already knows the answer from its training, RAG adds latency for nothing. And if your real problem is that the model needs to do things rather than know things, you want tools and agents, not retrieval. RAG is specifically for grounding answers in a body of text too large or too fresh to fit in the prompt. When that is your problem it is the right tool; when it is not, it is expensive scaffolding around a question you could have answered more simply.

Used where it fits, RAG is one of the most useful patterns in applied AI. Used everywhere by reflex, it is a common source of needless complexity. The skill is not building RAG — the code above is most of it — the skill is knowing when the problem in front of you actually calls for it.

Chunking is where most RAG systems are won or lost

If retrieval is where RAG usually fails, chunking is why. How you split your documents determines what can ever be retrieved. Chunk too large and each piece contains several unrelated ideas, so the relevant sentence arrives buried in noise that dilutes the answer. Chunk too small and you sever the context a passage needs to make sense, retrieving a fragment that is technically relevant but useless on its own. The sweet spot depends on your content, and finding it is empirical: try a few chunk sizes, run your real questions through, and look at what actually gets retrieved. Overlapping chunks — letting each piece share a little text with its neighbours — helps ensure an idea that straddles a boundary is not lost. This unglamorous preprocessing step has more impact on answer quality than any amount of prompt engineering, which is exactly why it is the step most tutorials rush past.

Grounding, and why it matters more than cleverness

The quiet superpower of RAG is not that it makes the model smarter — it is that it makes the model accountable. When you instruct the model to answer only from the retrieved passages, you can show the user which passages the answer came from. That citation turns an unverifiable claim into a checkable one, which is the difference between a toy and a system a business will trust. A support agent that says "your refund window is 30 days, per section 4 of the returns policy" and links the source is fundamentally more useful than one that simply asserts a number it may have invented. This is why RAG dominates enterprise AI: not because retrieval is clever, but because grounding answers in citable sources is what makes generative AI safe to put in front of customers. Build RAG for the grounding, treat the retrieval quality as the thing that matters, and you will have something far more valuable than a chatbot that sounds confident.