Fully AI generated notes.
1. Why Does RAG Exist?
An LLM primarily carries knowledge in its parameters.
During training:
Training Data
↓
Model Training
↓
Weights / Parameters
↓
Parametric Knowledge
Once trained, asking the model a question does not normally cause it to search through its original training documents.
The answer is generated from what has been learned into the model’s parameters.
That creates obvious limitations:
- knowledge can become stale
- private organizational data was never part of training
- the model may not know niche information
- updating knowledge by retraining is expensive
- generated answers aren’t inherently grounded in a specific source
RAG — Retrieval-Augmented Generation — adds external knowledge.
USER QUERY
│
▼
RETRIEVER
│
Search knowledge
│
▼
Relevant documents
│
▼
Query + retrieved context
│
▼
LLM
│
▼
ANSWER
This creates a useful separation:
LLM Parameters
↓
Parametric Memory
External Documents
↓
Non-Parametric / External Memory
The interesting problem then becomes:
Given a query and potentially millions of documents, how do we find the right information?
That takes us into information retrieval.
2. The Simplest Retrieval: Word Matching
Imagine a corpus:
D1: Java supports concurrency
D2: Python is a programming language
D3: Java virtual threads improve concurrency
And the query:
"Java concurrency"
The simplest idea is:
Find documents containing the query words.
This leads to Bag of Words representations.
Conceptually:
Vocabulary:
Java
Python
concurrency
virtual
threads
programming
A document can be represented by which vocabulary terms occur within it.
Because a document contains only a tiny fraction of the entire vocabulary, most dimensions are zero.
Hence:
Sparse representation
And retrieval based on these representations becomes:
Sparse retrieval
3. Term Frequency — TF
Not every word appearing in a document is equally important.
One simple signal is:
How often does this term occur inside this document?
That’s Term Frequency (TF).
Suppose:
D1:
"Java Java Java concurrency programming"
Then:
TF(Java) = high
TF(concurrency) = lower
TF answers:
How important/frequent is this term within this particular document?
But TF alone has a problem.
Words appearing everywhere aren’t necessarily useful for identifying relevant documents.
4. Inverse Document Frequency — IDF
Suppose our corpus contains 1 million documents.
If:
"programming"
appears in 800,000 documents, seeing it tells us relatively little.
But:
"virtual-thread-pinning"
might occur in only 500 documents.
That term is much more discriminative.
This gives us Inverse Document Frequency (IDF).
Conceptually:
Appears in many documents
↓
Common term
↓
Low IDF
Appears in few documents
↓
Rare term
↓
High IDF
A common formulation is:
[ IDF(t)=\log\frac{N}{df(t)} ]
where:
N = number of documents
df(t) = number of documents containing term t
A critical distinction:
TF
↓
How many times does the term occur
inside THIS document?
DF
↓
How many DOCUMENTS contain the term?
If:
D1 = "Java Java Java Java"
then:
TF(Java, D1) = 4
but
DF contribution = 1 document
We don’t count the same document four times when calculating document frequency.
5. TF-IDF
Now combine the two ideas:
[ TF\text{-}IDF = TF \times IDF ]
A term becomes important when it is:
Frequent in THIS document
+
Rare across ALL documents
This provides a much better relevance signal than raw word counts.
The evolution so far:
Bag of Words
↓
Term Frequency
↓
Inverse Document Frequency
↓
TF-IDF
6. BM25
BM25 builds upon the same general lexical-retrieval intuition.
Suppose:
Query:
"Java virtual threads"
For a document, BM25 effectively evaluates the contribution of the query terms:
score(Java)
+
score(virtual)
+
score(threads)
↓
Document relevance score
But BM25 improves upon simple TF-IDF in important ways.
Term-frequency saturation
Suppose:
D1:
"Java Java Java Java Java Java Java..."
Seven occurrences should not necessarily make the document seven times more relevant than one occurrence.
BM25 makes additional occurrences progressively less valuable.
Conceptually:
Relevance
│
│ _________
│ __/
│ __/
│ __/
│__/
└──────────────────────
Term Frequency
Document-length normalization
A 10,000-word document naturally has more opportunities to contain a term than a 100-word document.
BM25 accounts for document length.
Rare terms remain important
BM25 also incorporates an IDF-like notion.
So a rare query term can contribute more relevance than an extremely common one.
7. Sparse Retrieval
TF-IDF and BM25 belong to the sparse retrieval family.
The mental model:
Query:
"Java concurrency"
Document:
"Java concurrency with virtual threads"
↓
Strong lexical overlap
↓
High relevance
Sparse retrieval is particularly good when exact terminology matters:
Product IDs
Error codes
Names
Technical terms
Acronyms
Exact phrases
But there is an obvious weakness.
Consider:
Query:
"How do I run multiple tasks simultaneously in Java?"
Document:
"Java concurrency using virtual threads"
The meanings are highly related.
But the words don’t overlap very much.
This takes us to dense retrieval.
8. Dense Retrieval
Instead of representing text using vocabulary dimensions, an embedding model converts text into a dense numerical vector.
"Java concurrency"
↓
Embedding Model
↓
[0.21, -0.81, 0.17, 0.42, ...]
Likewise:
"Executing multiple tasks simultaneously"
↓
Embedding Model
↓
[0.19, -0.79, 0.21, 0.39, ...]
These vectors can be close even though the original text uses different words.
Now retrieval asks:
Which document vectors are closest to the query vector?
rather than:
Which documents contain the same words?
So:
SPARSE RETRIEVAL
Query
↓
Words
↓
Word overlap
↓
Relevant documents
DENSE RETRIEVAL
Query
↓
Embedding
↓
Vector similarity
↓
Semantically similar documents
9. Cross-Encoder Retrieval
One way to calculate relevance is to give the model both the query and document together.
Query ───────┐
│
├── Transformer
│ ↓
Document ────┘ Relevance Score
Because the model sees both simultaneously, it can model very detailed interactions between them.
This can provide excellent relevance scoring.
But imagine having:
10,000,000 documents
For every query:
Query + D1 → Model
Query + D2 → Model
Query + D3 → Model
...
Query + D10,000,000 → Model
That is far too expensive for first-stage retrieval.
10. Bi-Encoder / Dual-Encoder Retrieval
Instead, encode the query and documents independently.
Documents can be embedded ahead of time:
D1 → Encoder → Vector 1
D2 → Encoder → Vector 2
D3 → Encoder → Vector 3
...
Store those vectors.
Then at query time:
Query
↓
Encoder
↓
Query Vector
↓
Find nearby document vectors
This is dramatically more scalable.
The trade-off:
Query and document don’t interact deeply while their representations are being produced.
Their interaction occurs through vector similarity afterward.
This can lose some fine-grained relevance information compared with a cross-encoder.
11. ColBERT — A Middle Ground
A normal bi-encoder might reduce an entire query to:
Query → ONE vector
and an entire document to:
Document → ONE vector
ColBERT retains token-level representations.
For:
Query:
"Java concurrency"
we might conceptually retain:
Java → vector Q1
concurrency → vector Q2
And for a document:
"Java supports concurrent programming"
retain:
Java → D1
supports → D2
concurrent → D3
programming → D4
For each query-token representation, ColBERT finds the best matching document-token representation.
Java ─────────────→ Java
concurrency ──────→ concurrent
This MaxSim interaction preserves more fine-grained matching than collapsing everything into a single vector.
The trade-off is increased storage and retrieval complexity.
12. The Scaling Problem
Dense retrieval now gives us another problem.
Suppose we have:
10,000,000 document vectors
A query arrives.
Naively:
Query Vector
↓
Compare with D1
Compare with D2
Compare with D3
...
Compare with D10,000,000
That’s an exhaustive nearest-neighbor search.
Conceptually:
[ O(N) ]
As the vector collection becomes huge, comparing against everything becomes expensive.
This leads to:
Approximate Nearest Neighbor Search — ANN
Instead of demanding the mathematically exact nearest neighbors:
Find very likely nearest neighbors much faster.
This trades some retrieval recall for enormous improvements in latency and scalability.
Three important names appear here:
ANN
│
├── IVF
├── HNSW
└── PQ
But they solve somewhat different problems.
13. IVF — Reduce the Search Space
IVF — Inverted File Index
The core idea:
Don’t search the entire vector space. First determine which regions are promising.
During indexing, run something like K-means over the document vectors.
VECTOR SPACE
Cluster A Cluster B
• • • • • •
• • • • • • • •
• • • •
Cluster C
• • •
• • • • •
• •
Each cluster has a centroid.
Documents are assigned to nearby clusters.
At query time:
Query
↓
Find nearby cluster centroids
↓
Search only those clusters
↓
Find nearest document vectors
Instead of:
Search 10,000,000 vectors
we might search only a fraction of them.
The key mental model:
IVF reduces WHERE we search.
14. HNSW — Navigate Instead of Scan
HNSW — Hierarchical Navigable Small World
HNSW takes a completely different approach.
Vectors become nodes in a graph.
Nearby vectors have graph connections.
A ───── B
│ │
│ │
C ───── D ───── E
\
F
Instead of scanning every vector, search navigates through this graph toward increasingly promising neighbors.
HNSW adds hierarchical layers.
Sparse upper layer
↓
Fast large jumps
↓
More detailed layer
↓
Smaller jumps
↓
Dense bottom layer
↓
Nearest neighbors
Conceptually:
Start somewhere
↓
Move toward query
↓
Get closer
↓
Descend layer
↓
Refine search
↓
Nearest candidates
The key mental model:
HNSW reduces search work by navigating a graph rather than scanning the whole vector collection.
15. Product Quantization — Reduce Storage
Now consider another problem.
Suppose there are:
10,000,000 vectors
Each vector has:
1024 dimensions
And each dimension is:
float32 = 4 bytes
One vector therefore needs:
[ 1024 \times 4 = 4096 \text{ bytes} ]
For 10 million vectors:
[ 10,000,000 \times 4096
40.96\text{ GB} ]
That’s a substantial amount of memory just for the vectors.
This is where Product Quantization (PQ) enters.
The important idea—not the implementation details—is:
Approximate large vectors using compact codes rather than storing every floating-point value.
Suppose:
1024-D vector
is split into:
16 sub-vectors
Each sub-vector contains:
1024 / 16 = 64 dimensions
For each sub-vector position, K-means learns representative vectors called centroids.
Then instead of storing:
[0.23, 0.91, -0.72, 0.14, ...]
we store:
"this sub-vector looks most like centroid #17"
Across 16 sub-vectors, the original vector becomes something conceptually like:
[17, 203, 4, 91, 7, ..., 28]
These centroid IDs are the PQ codes.
If each subspace has:
256 possible centroids
then a centroid ID requires:
8 bits = 1 byte
Therefore:
16 sub-vectors
×
1 byte
=
16 bytes/vector
instead of:
4096 bytes/vector
For the example above:
Original vectors:
≈ 40.96 GB
PQ codes:
≈ 160 MB
That’s an enormous reduction.
But information has been discarded.
So distance calculations become approximate.
The key mental model:
PQ reduces HOW MUCH vector data we store and process, at the cost of precision.
The details of codebooks, lookup tables, asymmetric distance computation and exact scoring can be revisited when vector-database internals become important.
16. IVF vs HNSW vs PQ
This distinction is worth remembering.
IVF
↓
Partition the vector space
↓
Search fewer regions
HNSW
↓
Build a navigable graph
↓
Reach nearby vectors quickly
PQ
↓
Compress vectors
↓
Store and compare cheaper approximations
So they aren’t simply three competing versions of the same algorithm.
They attack different scaling problems.
And techniques can be combined.
For example:
IVF
↓
Find promising regions
+
PQ
↓
Store vectors compactly
=
IVF-PQ
17. Retrieval Accuracy vs Efficiency
Approximate retrieval introduces an important system-design trade-off.
Exact search:
High accuracy
↓
High compute / memory cost
Approximate search:
Much faster / smaller
↓
Potentially miss some good candidates
This is why retrieval systems often care about metrics such as Recall@K.
Suppose exact nearest-neighbor search identifies the true best 100 documents.
Approximate retrieval finds 94 of them.
Recall@100 = 94%
Whether that’s acceptable depends on the application.
For many RAG systems, first-stage retrieval doesn’t need perfect ordering.
It needs to ensure:
The genuinely useful documents survive into the candidate set.
18. Reranking
This brings us back to the cross-encoder.
Earlier, cross-encoders were too expensive:
Query × 10,000,000 documents
But what if dense retrieval first reduces the candidates?
10,000,000 documents
↓
ANN retrieval
↓
Top 100 candidates
↓
Cross-encoder
↓
Top 10
Now expensive query-document interaction becomes practical.
This creates a common retrieval architecture:
QUERY
│
▼
Fast Retriever
│
millions of docs
│
▼
Top Candidates
│
▼
Reranker
│
deeper Q-D interaction
│
▼
Best Documents
The first stage optimizes:
Recall + speed
The second stage optimizes:
Precision
19. Sparse + Dense = Hybrid Retrieval
Sparse and dense retrieval have complementary strengths.
Consider:
"ERR_JAVA_4821"
An embedding model may not understand this identifier particularly well.
BM25 can match it exactly.
Conversely:
"How do I execute many lightweight tasks simultaneously?"
may semantically match:
"Java virtual thread concurrency"
even with weak lexical overlap.
Dense retrieval is useful here.
Therefore modern RAG systems frequently combine both.
QUERY
│
┌────────────┴────────────┐
│ │
▼ ▼
Sparse Search Dense Search
BM25 Embeddings
│ │
▼ ▼
Exact lexical Semantic
matches matches
│ │
└────────────┬────────────┘
│
▼
Merge Candidates
│
▼
Reranker
│
▼
Best Documents
│
▼
LLM
This is hybrid retrieval.
20. Retrieval and Generation Are Separate Problems
One of the most useful RAG distinctions is:
RAG
│
┌─────────┴─────────┐
│ │
RETRIEVAL GENERATION
│ │
"What information "What should I
should the model say using that
receive?" information?"
A bad RAG answer does not automatically mean the LLM failed.
Suppose:
User Query
↓
Retriever chooses wrong chunks
↓
Perfectly capable LLM
↓
Wrong / incomplete answer
The generation model never received the information it needed.
Conversely:
User Query
↓
Excellent retrieval
↓
Correct documents
↓
Poor generation/reasoning
↓
Bad answer
So RAG quality must be understood as at least two separate systems:
Retrieval Quality
+
Generation Quality
=
Overall RAG Quality
21. The Complete Retrieval Progression
The entire story now fits together.
INFORMATION RETRIEVAL
│
▼
Bag of Words
│
▼
TF-IDF
│
▼
BM25
│
▼
SPARSE RETRIEVAL
│
Exact lexical matching
│
│
▼
DENSE RETRIEVAL
│
Embeddings
│
Semantic similarity search
│
┌────────────┴────────────┐
│ │
▼ ▼
Bi-Encoder ColBERT
one vector token vectors
│ │
▼ │
Large-scale │
vector search │
│ │
▼ │
ANN ◄───────────────────────┘
│
┌──────────┼──────────┐
│ │ │
▼ ▼ ▼
IVF HNSW PQ
│ │ │
reduce graph compress
search navigation vectors
space
│
└──────────┬──────────┘
│
▼
Candidate Documents
│
▼
Reranker
│
▼
Best Documents
│
▼
Add to LLM Context
│
▼
LLM
│
▼
ANSWER
22. What Is Actually Worth Remembering?
The implementation details can disappear from memory.
The structure shouldn’t.
Sparse vs Dense
Sparse
→ words matching words
→ TF-IDF / BM25
Dense
→ meaning matching meaning
→ embeddings
Bi-encoder vs Cross-encoder
Bi-encoder
→ encode query and documents independently
→ scalable
Cross-encoder
→ query + document together
→ richer interaction
→ expensive
→ excellent for reranking
ANN
Millions of vectors
↓
Exact comparison becomes expensive
↓
Approximate Nearest Neighbor search
IVF
Reduce where I search.
HNSW
Navigate efficiently toward nearby vectors.
PQ
Compress what I store.
Reranking
Retrieve broadly and cheaply first; evaluate a small candidate set more carefully afterward.
Hybrid retrieval
Use lexical and semantic retrieval together because each catches things the other misses.
23. The RAG Mental Model
When everything else is forgotten, reconstruct it from this:
USER QUERY
│
▼
Understand the query
│
▼
RETRIEVAL
│
┌─────────────┴─────────────┐
▼ ▼
BM25 Embeddings
Sparse Dense
│ │
└─────────────┬─────────────┘
▼
ANN at scale
│
┌────────────┼────────────┐
▼ ▼ ▼
IVF HNSW PQ
│
▼
Candidate chunks
│
▼
Rerank
│
▼
Best relevant chunks
│
▼
Query + RAG Context
│
▼
LLM
│
▼
ANSWER
RAG itself is simple:
Retrieve useful information before asking the model to generate.
The complexity comes from making “retrieve useful information” work accurately, quickly and cheaply at scale.
And that is why concepts that existed long before LLMs—TF-IDF, BM25, inverted indexes, nearest-neighbor search, clustering, quantization and information retrieval—suddenly become part of understanding modern generative AI systems.