“Chat with your PDF”, a company help bot that knows the latest policy, an AI tutor that answers from your syllabus: almost all of these run on one technique called RAG, short for retrieval-augmented generation. It's also one of the most in-demand AI skills in job listings right now, and the idea is simple enough to understand in ten minutes.
The problem RAG solves
A language model like the one inside ChatGPT or Claude has two big blind spots:
- It only knows its training data, frozen at a cut-off date. Last week's notice isn't in there.
- It has never seen your private documents: your class notes, your company's handbook, your PDFs.
Ask it about either and it will often guess, confidently. RAG fixes this the way a good student handles an open-book exam: find the right page first, then write the answer.
How RAG works in 3 steps
- Retrieve: search your documents for the passages most relevant to the question.
- Augment: paste those passages into the prompt, together with the question.
- Generate: the model writes an answer using the passages, and cites them.
Try it with a (made-up) college handbook. Switch between “With RAG” and “Without RAG”:
🎮 Try RAG on a college handbook
① Retrieve: relevance of each chunk
- §4.2 Attendance2
Students must maintain at least 75% attendance in every subject to be eligible for the end-semester exam.
- §4.3 Medical leave1
Up to 10% attendance shortfall may be condoned with a medical certificate submitted within 7 days of returning.
- §6.1 Exam fees0
- §7.4 Hostel mess0
- §8.2 Library0
- §9.1 Re-evaluation0
② Augment: what the AI actually receives
Answer ONLY from the passages below. If the answer isn't there, say you don't know. [1] §4.2 Attendance: Students must maintain at least 75% attendance in every subject to be eligible for the end-semester exam. [2] §4.3 Medical leave: Up to 10% attendance shortfall may be condoned with a medical certificate submitted within 7 days of returning. Question: What is the minimum attendance?
③ Generate: the answer
You need at least 75% attendance in every subject to sit the end-semester exam (Handbook §4.2).
This demo scores chunks by matching words. Real RAG systems compare meanings with embeddings, explained below.
Behind the scenes: chunks, embeddings and vector search
Real RAG systems prepare documents once, before any questions arrive:
- Chunking: split long documents into small passages (a few paragraphs each), so only the useful parts get sent to the model.
- Embedding: turn each chunk into an embedding, a list of numbers that captures its meaning.
- Storing: save the embeddings in a vector database that can quickly find “nearby” meanings.
At question time, the question is embedded too, and the database returns the chunks whose meaning is closest. That is why a RAG bot can match “Can I bunk classes?” to a passage about attendance rules, even though the words are different.
Build a tiny RAG system (Python or JavaScript)
This is RAG at its smallest: a word-matching retriever plus one call to Claude through the official Anthropic SDK. Swap the retrieve function for an embedding search when you have more than a handful of documents.
# A tiny RAG system: retrieve the best passages, then let Claude answer from them.
# pip install anthropic and set the ANTHROPIC_API_KEY environment variable
import anthropic
client = anthropic.Anthropic()
DOCS = [
"Students must maintain at least 75% attendance in every subject.",
"A medical certificate can cover up to a 10% attendance shortfall.",
"The exam fee is Rs 1,500, due by the 15th; a Rs 500 late fee applies after.",
"The hostel mess serves dinner from 7:30 to 9:30 pm.",
]
def retrieve(question, k=2):
"""Score each passage by shared words (real systems use embeddings)."""
q_words = set(question.lower().split())
scored = sorted(DOCS, key=lambda d: len(q_words & set(d.lower().split())), reverse=True)
return scored[:k]
def ask(question):
passages = retrieve(question)
context = "\n".join(f"[{i + 1}] {p}" for i, p in enumerate(passages))
response = client.beta.messages.create(
model="claude-opus-5-5",
max_tokens=4096,
# If the model declines on a safety policy, retry on a fallback model automatically
betas=["server-side-fallback-2026-07-01"],
fallbacks="default",
system=(
"Answer ONLY from the numbered passages. Cite them like [1]. "
"If the answer is not in the passages, say you don't know."
),
messages=[{"role": "user", "content": f"Passages:\n{context}\n\nQuestion: {question}"}],
)
if response.stop_reason == "refusal":
return "Sorry, I can't help with that one."
return "".join(block.text for block in response.content if block.type == "text")
print(ask("What is the minimum attendance?"))
# e.g. "You need at least 75% attendance in every subject [1]."RAG vs fine-tuning
| Feature | RAG | Fine-tuning |
|---|---|---|
| What it changes | What the model reads at question time | The model's weights (it is retrained) |
| Best for | Facts, documents, policies, anything that changes | Style, tone, format, specialised skills |
| Updating knowledge | Swap or add documents: minutes | Train again: hours to days, and money |
| Can cite sources? | Yes, the passages it used | No |
| Cost to start | Low | Higher |
| Typical example | “Chat with your PDF”, company help bots | A model that always answers in your brand's voice |
Where RAG is used
- “Chat with your PDF” tools and AI features in note-taking apps.
- Customer support bots that answer from help-centre articles.
- Company assistants that search internal wikis, policies and tickets.
- AI search engines that read web pages and cite them.
- Study tools that quiz you from your own syllabus.
Common RAG problems (and fixes)
- Wrong passages retrieved: improve chunking, use better embeddings, or retrieve more passages.
- Answer not in the documents: instruct the model to say so instead of guessing.
- Outdated documents: RAG is only as good as what it reads. Keep the sources fresh.
- Too much text: sending whole books wastes money and can bury the key line. Send the best few chunks.
Where to go next
RAG lets AI read before it answers. The next step is letting it act: searching, clicking and using tools on its own. That's the next part of the course: what is an AI agent?
Key takeaways
- ✓RAG = retrieve relevant passages, add them to the prompt, then generate the answer.
- ✓It gives AI fresh and private knowledge without retraining the model.
- ✓Embeddings and vector databases find passages by meaning, not just matching words.
- ✓Tell the model to answer only from the passages, cite them, and admit when it doesn't know.
- ✓Use RAG for facts that change; use fine-tuning for style and behaviour.
Frequently asked questions
What is RAG in AI?
RAG stands for retrieval-augmented generation. Before a language model answers, the system retrieves the most relevant passages from a set of documents (your notes, a company wiki, a textbook) and adds them to the prompt. The model then writes its answer from those passages, so it can use information it was never trained on and cite where the answer came from.
Why is RAG useful?
Language models only know what was in their training data, which is frozen at a cut-off date and doesn't include your private documents. RAG gives them fresh, specific, private information at question time, reduces hallucinations because the model has real text to work from, and lets you check answers against the cited sources.
What is the difference between RAG and fine-tuning?
RAG gives the model information at question time by looking it up; fine-tuning changes the model itself by training it further on examples. RAG is best for facts that change or must be cited (policies, documentation, notes). Fine-tuning is better for teaching a consistent style, format or specialised behaviour. Many real systems use RAG first because it is cheaper and easy to update.
What are embeddings and vector databases?
An embedding is a list of numbers that represents the meaning of a piece of text, so texts with similar meanings get similar numbers. A vector database stores these embeddings and can quickly find the chunks closest in meaning to a question, even when they don't share the same words. Together they power the “retrieve” step in most RAG systems.
Does RAG completely stop hallucinations?
No, but it reduces them a lot. If retrieval finds the wrong passages, or the answer isn't in the documents, the model can still guess. Good RAG systems tell the model to answer only from the passages, to say when the answer isn't there, and to cite its sources so a human can check.
🧠 Quick check: RAG
1. What does the “retrieval” in RAG do?
2. Your college handbook changes every semester. What's the simplest way to keep a chatbot up to date?
3. Why do RAG systems use embeddings instead of only matching keywords?
4. What should a good RAG prompt tell the model to do if the answer isn't in the passages?
Checking your access…