RAG vs CAG vs KAG: How Each Works, What the Research Says and Which One to Use

Large language models have a knowledge problem. They only know what was in their training data. Ask about something that happened after training, or about information that was never public, like your college's fee circular or a company's internal policy, and they either say they don't know or, worse, make something up.

Three approaches have become the standard ways to fix this: RAG, CAG and KAG. If you're learning AI or building with it, the RAG vs CAG vs KAG question is one you'll face again and again. They sound similar, but they solve the knowledge problem in very different ways.

In this guide I explain all three in plain English, look at what the original research papers actually found, and run a small hands-on experiment that shows where each one wins and fails. Here's what's covered:

• what RAG, CAG and KAG are, with simple analogies

• how each one works, step by step

• what the CAG and KAG research papers really measured

• a hands-on test where plain RAG got a multi-hop question wrong and a knowledge graph got it right

• a cost comparison that surprised me

• a simple decision guide, plus how teams combine all three

RAG vs CAG vs KAG: the quick answer

If you only read one section, read this one:

What is RAG (retrieval-augmented generation)?

RAG was introduced in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research (now Meta), University College London and New York University. Today it's the most common way to connect an LLM to your own data.

The idea: don't make the model remember everything. When a question comes in, search your documents for the most relevant pieces and hand them to the model along with the question.

As IBM's explainer video on the topic describes, RAG works in two phases:

Offline phase (done once, then updated): 1. Take your documents: PDFs, web pages, notes and manuals. 2. Split them into small chunks. 3. Convert each chunk into an embedding, a list of numbers that captures its meaning. 4. Store the embeddings in a vector database.

Online phase (every question): 1. Convert the user's question into an embedding with the same model. 2. Find the top few most similar chunks, often three to five. 3. Put those chunks into the prompt, next to the question. 4. The LLM writes an answer using that context, ideally citing its sources.

Why people like it: - Freshness: add or remove documents and the very next answer reflects it, with no retraining. - Scale: you can index millions of documents, because the model only ever sees a few chunks at a time. - Citations: you know exactly which chunks the answer came from. - Modularity: you can swap the embedding model, vector database or LLM independently.

Where it struggles: RAG is only as good as its retriever. If the search picks the wrong chunks, the model either can't answer or answers confidently from the wrong context. Chunking, metadata and ranking all need care. As NovelVista puts it, "RAG is not just a vector database project."

If you want to see how far vector search engineering can go, our deep dive on how the LEANN vector index shrank a 201 GB index to 6 GB covers the storage side of RAG.

What is CAG (cache-augmented generation)?

CAG comes from a December 2024 paper with a provocative title: "Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks." It was written by Brian J. Chan, Chao-Ting Chen and Jui-Hung Cheng of National Chengchi University and Hen-Hsen Huang of Academia Sinica, both in Taiwan.

The idea: modern models have huge context windows. If your knowledge base is small enough, why search at all? Load the entire knowledge base into the model once, save the result, and reuse it for every question.

Here's how it works:

1. Preload: put all your documents into the model's context in one go.

2. Cache: as the model reads that text, it builds an internal memory called the KV cache (key-value cache) in every attention layer. CAG saves this cache to memory or disk.

3. Answer: for each new question, load the saved cache, add the question and generate. The model doesn't need to re-read the documents.

4. Reset: after answering, trim off the question's tokens so the cache is clean for the next one.

Think of it as a student who reads the whole textbook the night before the exam and keeps it all in mind, instead of flipping to the index for every question.

Why people like it: - No retrieval step: no vector database, no chunking decisions and no risk of fetching the wrong chunk. - Speed: there's no search, and the documents don't need to be reprocessed each time. - Completeness: the model sees everything, so nothing relevant is left out.

Where it struggles: - Size: everything must fit in the context window. That's a hard limit. - Freshness: when the documents change, the cache must be rebuilt. - Focus: with everything loaded, the model has to find the right detail itself, and it can get distracted by unrelated text.

What the CAG paper actually found

The researchers tested Llama 3.1 8B, which supports 128,000 tokens of context, on two question-answering benchmarks, SQuAD and HotpotQA. They used three knowledge-base sizes, from about 21,000 to 85,000 tokens. They compared CAG with RAG using two retrievers, keyword-based BM25 and dense embeddings, scored with BERTScore.

• Accuracy: CAG scored highest in all six test setups. But the margins were small, often under 0.01. On large HotpotQA, for example, CAG scored 0.7527 against 0.7495 for the best RAG setting.

• Speed: compared with feeding the same documents in as a normal long prompt every time, CAG was dramatically faster. On large HotpotQA, generation took 2.33 seconds with CAG versus 94.35 seconds without the cache.

Two caveats are worth knowing: - That speed comparison is CAG versus re-reading everything each time, not CAG versus RAG. The paper says CAG is faster than RAG too, but it doesn't publish RAG timing numbers. - It's one small model on two benchmarks with knowledge bases under 100,000 tokens. The authors themselves suggest a hybrid, where a base set of knowledge is preloaded and retrieval is used for edge cases.

CAG in the real world: prompt caching

You don't need to manage KV caches yourself to use the CAG idea. Major AI APIs, including Anthropic's and OpenAI's, offer prompt caching: if the beginning of your prompt is the same across requests, the provider stores it and charges much less for reusing it. According to Firecrawl's October 2026 comparison, cached reads are typically billed at around 10% of the normal input price on both, with a cache lifetime of minutes (Anthropic also offers a one-hour option).

That makes "preload the manual, then answer many questions" practical even for small teams.

What is KAG (knowledge-augmented generation)?

KAG comes from Ant Group, the company behind Alipay. Its 2024 paper, "KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation," was written by Ant Group's Knowledge Graph Team with researchers from Zhejiang University. The framework is open source on GitHub under the Apache 2.0 license and built on Ant's OpenSPG knowledge graph engine.

The idea: many real questions aren't "find the paragraph that looks similar." They're "connect these facts." KAG turns documents into a knowledge graph, a network of entities (people, products, rules, places) and the relationships between them, and links each fact back to the original text. Then it answers questions by planning logical steps and following those relationships.

Think of the difference like this. RAG asks "which paragraphs look most like this question?" KAG asks "which facts are connected to this one, and how?"

The paper explains the motivation directly. In vector search, similarity doesn't always match the relevance needed for reasoning, and plain RAG handles numbers, time relationships and expert rules poorly. KAG's five main components are:

1. LLM-friendly knowledge representation: facts stored in a way both graphs and LLMs can use

2. Mutual indexing: graph facts linked to the original text chunks, and back again

3. A logical-form-guided reasoning engine: complex questions broken into steps the system can execute

4. Knowledge alignment: cleaning up and connecting extracted facts using semantic reasoning

5. Model enhancements: improving the LLM's own abilities for these tasks

What the KAG paper actually found

The paper tested KAG on three multi-hop question benchmarks, where answering requires connecting several facts. Compared with HippoRAG, another graph-based method, it reports F1 score improvements of:

• 19.6% on 2WikiMultiHopQA

• 12.2% on MuSiQue

• 12.5% on HotpotQA

The abstract separately mentions a 33.5% relative improvement on HotpotQA without naming the baseline, so I'd treat the per-benchmark numbers above as the clearer comparison.

Ant Group also applied KAG to two real services, an e-government Q&A system and an e-health Q&A system, and reports better accuracy and professionalism than RAG. The paper doesn't give specific accuracy figures for those.

Where KAG struggles: building a good knowledge graph takes effort, and often domain experts. The graph also has to be kept up to date. The KAG project has been working on this. Its version 0.7 release in April 2025 added a "lightweight build" mode that it says cuts construction token costs by 89%.

A quick note on names: "KAG" is mostly associated with Ant Group's framework. Microsoft's GraphRAG, published in 2024, is a related but separate approach that uses LLM-built knowledge graphs and community summaries to answer broad, big-picture questions about a document collection.

Hands-on: I tested RAG and a knowledge graph on the same question

Reading about multi-hop reasoning is one thing. Seeing it fail is more convincing. So I built a tiny fictional college knowledge base with 12 one-sentence facts and tested two questions. I used a simple TF-IDF retriever as the "R" in RAG, and a small graph built with Python's NetworkX library as the KAG-style approach. No LLM was needed, because I'm only checking what each method would hand to the model.

The key facts were spread across four separate sentences:

• Dr. Meera Iyer won the 2026 Best Teacher Award.

• Dr. Meera Iyer teaches Data Structures.

• Data Structures classes are held in Room 204.

• Room 204 is in the Raman Block.

Question 1 (single-hop): "When must exam fees be paid?"

RAG nailed it. The top result was "Exam fees for the winter semester must be paid by 20 November," with a clear similarity score of 0.667. Everything else scored zero.

Question 2 (multi-hop): "In which block is the class taught by the 2026 Best Teacher Award winner held?"

This is where plain RAG broke down:

Even with the top five chunks, RAG never found "Dr. Meera Iyer teaches Data Structures" or "Room 204 is in the Raman Block." Those sentences don't mention the award, so they don't look similar to the question.

It got worse. In the top five, RAG pulled in "Machine Learning classes are held in Room 310" and "Room 310 is located in the Bose Block." That's a distractor path. An LLM handed those chunks could easily answer Bose Block, which is wrong.

The knowledge graph followed the relationships instead:

Dr. Meera Iyer -> teaches -> Data Structures -> held_in -> Room 204 -> located_in -> Raman Block

The answer was Raman Block, with every step visible. That explainable reasoning path is exactly what KAG is designed for.

And CAG? This whole knowledge base is only about 160 tokens, so CAG could simply preload everything. The model would see all four facts and could connect them itself. That shows CAG's real strength: when the data is small, you skip the retrieval problem entirely.

Some honest caveats. TF-IDF is a basic keyword retriever. Dense embeddings and rerankers do better on many questions, but they hit the same wall here, because the second-hop facts simply don't resemble the question. Modern "agentic" or iterative RAG can fix this by retrieving, reading, then searching again. That works, but it's slower, and it's really adding a reasoning loop on top of RAG.

RAG vs CAG cost: the result that surprised me

A common claim is that CAG is "cheaper." With API prompt caching, that depends heavily on the size of your knowledge base. I ran a simple cost model, using an example price of $3 per million input tokens, cached reads billed at 10% of that, and RAG sending five 300-token chunks per question. I ignored the one-time cache write cost.

Input cost per 1,000 queries, using the example assumptions above.

What this shows: - Caching matters enormously. CAG with caching is about 10 times cheaper than re-sending the whole knowledge base with every question. - RAG's cost stays flat however big your data gets, because it only ever sends a few chunks. - CAG beats RAG on cost only for small knowledge bases, below about 15,000 tokens with these assumptions. Above that, CAG's real benefits are speed, simplicity and completeness, not price.

Your own numbers will depend on the model, the provider's pricing and your traffic. But the shape of the curve holds: CAG cost grows with the size of your knowledge, and RAG cost doesn't.

RAG vs CAG vs KAG: which should you choose?

Here's a simple way to decide, combining the advice from IBM, NovelVista and the research:

Choose CAG when: - the whole knowledge base fits comfortably in the model's context window - it rarely changes, for example a product manual updated a few times a year - lots of people ask similar questions and speed matters - you want the simplest possible setup

Choose RAG when: - your data is large, from thousands to millions of documents - it changes often, like news, tickets, policies or product updates - you need citations to exact sources - you want a flexible, well-supported starting point (this is most projects)

Choose KAG when: - answers require connecting several facts or following rules - relationships matter: who owns what, what depends on what, which rule applies - explainability and traceability are required, as in healthcare, law, finance or government - you can invest in building and maintaining a knowledge graph

Regulated industries often end up leaning toward KAG-style approaches for that traceability. Our look at AI adoption in regulated systems covers why auditability matters so much there.

Three scenarios to test yourself

IBM's video "RAG vs. CAG: Solving Knowledge Gaps in AI Models" ends with a fun "RAG or CAG?" game. It's worth watching in full:

Here's my version of the game, with all three options:

1. An IT helpdesk bot built on a 200-page product manual that's updated twice a year. Answer: CAG. The manual fits in a modern context window, rarely changes, and users want fast answers.

2. A research assistant for a law firm searching thousands of constantly updated cases, with exact citations. Answer: RAG. The data is huge and changing, and citations come naturally from retrieval.

3. A hospital system checking whether a drug is safe for a patient, given their conditions, current medicines and treatment guidelines. Answer: KAG, combined with RAG. The answer depends on following relationships between drugs, conditions and interactions, and it must be explainable to the doctor. RAG can still pull in the latest research papers.

The future of RAG vs CAG vs KAG is hybrid

In practice, mature systems rarely use just one approach. Common combinations include:

• CAG + RAG: preload the stable core, like FAQs and policies, and retrieve only for rare or new questions. The CAG paper's authors suggest this themselves. IBM's video describes a related pattern: use RAG to pull a patient's relevant records, then load them into a long-context model as a temporary "working memory" for follow-up questions.

• KAG + RAG: use the knowledge graph for reasoning and rules, and RAG for fresh documents. KAG's own design already mixes graph reasoning with text retrieval.

• Agentic RAG: an AI agent decides when to search, what to search for next and when to stop, so it handles multi-hop questions step by step. Our explainer on how agentic AI works covers that loop in detail.

My take

After digging into the papers and running these tests, here's how I see it.

RAG is still the right default for most projects, and it's the one every AI learner should build first. CAG isn't a replacement for RAG. It's a shortcut for small, stable knowledge, and with API prompt caching it's now trivially easy to try. KAG is the specialist: harder to build, but it's the only one of the three that truly reasons over connected facts and shows its work.

The bigger lesson from my small test is that retrieval quality is the real bottleneck, not the LLM. When the right facts didn't reach the model, no amount of model intelligence could fix the answer. Whichever approach you choose, measure what you're actually feeding the model.

The bottom line

RAG retrieves relevant chunks on demand and is best for large, changing data with citations. CAG preloads a small, stable knowledge base into the model's cache for fast, simple answers, and it's only cheaper than RAG when the knowledge is small. KAG builds a knowledge graph and reasons over relationships, which makes it best for multi-step, rule-heavy and explainable answers in specialist domains. The real question isn't RAG vs CAG vs KAG as rivals. It's which mix fits your data size, how often it changes and how much reasoning your questions need.

Sources:

• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020

• Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks, Chan, Chen, Cheng and Huang, arXiv, December 2024

• KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation, Ant Group Knowledge Graph Team, arXiv, 2024

• OpenSPG/KAG on GitHub, Apache 2.0

• Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," Microsoft Research, 2024

• Prompt caching: how it works, what it costs, Firecrawl, October 18, 2026

• RAG vs KAG vs CAG: Decoding the Future of AI-Augmented Language Models, Anandhu Krishna, Generative AI publication

• RAG vs CAG vs KAG: The AI Architecture Decision Every Tech Leader Will Face in 2026, NovelVista

• RAG vs. CAG: Solving Knowledge Gaps in AI Models, IBM Technology, YouTube