RAG Hallucination Benchmark
How often does a small RAG tutor invent answers when the knowledge base doesn't contain them? We built the test on our own open dataset docs, ran 90 generations through a 0.5B model with strict grounding instructions, and graded every one by hand. The answer: 30–70% of the time.
What the benchmark found
| Config | Citation accuracy | Refusal rate | Hallucination rate |
|---|---|---|---|
| A — chunk 500, top_k 3 | 4/10 (40%) | 3/10 (30%) | 7/10 (70%) |
| B — chunk 1000, top_k 3 | 5/10 (50%) | 4/10 (40%) | 5/10 (50%) |
| C — chunk 500, top_k 5 | 6/10 (60%) | 5/10 (50%) | 3/10 (30%) |
A 0.5B model, explicitly instructed to answer only from provided documents and to refuse when the answer is absent, still fabricated answers on 30–70% of the unanswerable questions. The fabrications were confident: fake citations, invented license restrictions, a refusal sentence followed by a fabricated answer in the same output.
The cheapest win was retrieval volume, not model size: raising top_k from 3 to 5 halved the hallucination rate (70% → 30%). Retrieval was never the bottleneck — the correct chunk was retrieved for all 10 answerable questions; the failures happened in generation. The quieter failure ran the other way: the model refused 4 answerable questions whose answers were in context.
Bottom line for anyone deploying a teaching RAG: a 0.5B stack is not safe for student-facing use. Test the unanswerable case — the case that harms students most — and budget for a larger generator or a verification layer. Prompt engineering alone did not solve it. The full research report is Dispatch #05.
How the benchmark was run
Corpus: the 5 Study Intelligence open dataset docs (README, methodology notes, changelog) — chunked at 500 chars (124 chunks) and 1000 chars (64 chunks). Embeddings: all-MiniLM-L6-v2 via ONNX Runtime, CPU-only. Generator: Qwen2.5-0.5B-Instruct Q4_K_M via llama-cpp, temperature 0.1, max 180 tokens — our proxy for a small self-hosted teaching model.
Test set: 30 questions — 10 answerable (fact stated explicitly in the corpus), 10 partial (needs synthesis), 10 unanswerable (plausible in-domain questions with no answer in the corpus, e.g. ACLAS's ROR ID while its application is still in progress). All 90 generations (30 × 3 configs) were graded manually, single-rater, with rule-based pre-grading.
- Single small model (0.5B) — larger models hallucinate less; the tradeoff curve is not measured here.
- 30 questions, one domain (AI-education dataset docs) — not a general RAG benchmark.
- Single-rater manual grading; no inter-rater reliability measured.
- No reranker, no hybrid retrieval — a production system would do better.
Frequently asked
What does "hallucination rate" mean in this benchmark?
The share of unanswerable questions — questions whose answer is absent from the knowledge base — for which the model invented an answer instead of refusing. Across three RAG configurations the rate ran 30–70%.
Why test a 0.5B model?
It is a proxy for a small self-hosted teaching deployment: cheap, CPU-only, and exactly the setup a resource-constrained institution would run. The benchmark is an honest stress test of that stack, not a showcase.
What was the cheapest fix for hallucinations?
Retrieving more context. Raising top_k from 3 to 5 halved the hallucination rate from 70% to 30% — cheaper than a bigger model, and retrieval itself was never the bottleneck.
Can I use this data?
Yes — CC-BY-4.0. Free for any use with attribution to Study Intelligence (aclas.si).
Get the data
Free under CC-BY-4.0. Cite as: Study Intelligence (2026). RAG Hallucination Benchmark for Teaching Knowledge Bases. https://aclas.si/data/rag-hallucination-benchmark
@dataset{studyintelligence2026raghallucination,
author = {{Study Intelligence}},
title = {RAG Hallucination Benchmark for Teaching Knowledge Bases},
year = {2026},
publisher = {Study Intelligence (aclas.si)},
url = {https://aclas.si/data/rag-hallucination-benchmark}
}Study Intelligence is a research initiative by ACLAS College.