RAG: what it is and how Retrieval-Augmented Generation works

yet, three structural weaknesses always accompany them:

1. Hallucinations: when they don't know, they make things up with total
   confidence.
2. Frozen knowledge: what they know ends at their training cutoff date.
3. Opacity: they cannot say where an answer came from or cite sources.

And a fourth one, the most important for any real project: they don't know your
data. Your internal documentation, your knowledge base, your support tickets,
your code. None of that lives in their weights.

What RAG is

RAG (Retrieval-Augmented Generation) is an architecture pattern that connects
an LLM with external knowledge sources: before generating an answer, the system
** retrieves** the relevant fragments from your corpus and injects them into
the prompt as context.

The formalization comes from Lewis et al. (Meta AI Research, 2020),
"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", although
today the term covers an entire family of architectures.

The classic analogy: a plain LLM is a student taking a closed-book exam from
memory. An LLM with RAG is the same student taking an open-book exam: they
don't need to memorize the manual, just know how to find the right page.

flowchart LR
    subgraph SIN["LLM without RAG — closed-book exam"]
        direction LR
        Q1[Question] --> M1[LLM]
        M1 --> R1["Answer without sources<br/>may hallucinate · frozen knowledge"]
    end
    subgraph CON["LLM with RAG — open-book exam"]
        direction LR
        Q2[Question] --> RET[Retrieval]
        RET --> CTX[Retrieved context]
        CTX --> M2[LLM]
        M2 --> R2["Traceable answer<br/>with citations and fresh data"]
    end

The three pillars

1. Retrieval

Given a corpus (documentation, PDFs, wikis, tickets, code), the system works in
two phases.

Offline phase (indexing):

flowchart LR
    A["Documents<br/>docs · PDFs · wikis · tickets · code"] --> B["Chunking<br/>fragments of 200–1000 tokens"]
    B --> C["Embedding model<br/>text → vector"]
    C --> D[("Vector database<br/>pgvector · Qdrant · Chroma · Pinecone")]

• Chunking: each document is split into manageable fragments (typically
  200–1000 tokens).
• Embeddings: a model converts each fragment into a vector representing its
  meaning.
• Vector store: pgvector, Qdrant, Chroma, Pinecone, Weaviate… store and index
  those vectors.

Online phase (query):

flowchart LR
    Q["User question"] --> E["Embedding of the question"]
    E --> S["Similarity search<br/>cosine distance"]
    V[("Vector store")] -. "top-k" .-> S
    S --> K["Top-k most<br/>relevant fragments"]

2. Augmentation

The retrieved fragments are inserted into a prompt with instructions like:
"answer only based on this context; if the information is not there, say you don't know."

3. Generation

The LLM generates the answer conditioned on the context. That enables citing
sources, reducing hallucinations, and answering with data it never saw during
training.

flowchart TB
    subgraph P1["1 · Retrieval"]
        direction TB
        subgraph OFF["Offline phase — indexing"]
            direction LR
            A[Documents] --> B[Chunking] --> C[Embeddings] --> D[("Vector store")]
        end
        subgraph ON["Online phase — query"]
            direction LR
            E[Question] --> F["Embedding of the question"] --> G["Similarity search"] --> H["Top-k fragments"]
        end
        D -.-> G
    end
    subgraph P2["2 · Augmentation"]
        I["Augmented prompt<br/>question + context + instructions"]
    end
    subgraph P3["3 · Generation"]
        J[LLM] --> K["Answer with citations<br/>and fresh data"]
    end
    H --> I
    I --> J

Why it matters

• Freshness: updating the index is updating the knowledge. Minutes, not
  retraining.
• Traceability: every claim can link to the exact fragment supporting it.
• Access control: retrieval can filter by metadata, tenant, or permissions.
• Cost: injecting a few thousand selected tokens per query is far cheaper
  than fine-tuning or stuffing the whole corpus into the context.

Real use cases: assistants over product documentation, support powered by
historical tickets, internal corporate search, legal or medical assistants with
scoped, auditable corpora.

In short

RAG doesn't make the model smarter: it gives it timely access to the right
information. And that phrase — the right information at the right time — is
exactly the topic of the next article: how RAG enriches search far beyond
keywords.