We are building a self-hosted, open-model tutoring stack: small teaching models plus RAG over open course materials. Before building it, we needed an honest number — how often does a small RAG system invent answers when the knowledge base doesn't contain them? Vendor benchmarks almost never test the unanswerable case. That is the exact case that harms students most: a confident, wrong answer presented as fact.
The experiment
We built a deliberately minimal RAG — the kind a small university could actually self-host. Corpus: our own five open dataset documents, chunked two ways (500 chars → 124 chunks, 1000 chars → 64 chunks). Embeddings: open, CPU. Retrieval: plain cosine similarity, no reranker. Generator: a 0.5B instruction-tuned model (Qwen2.5-0.5B-Instruct, quantized), our proxy for a small self-hosted teaching model — with a strict grounding prompt: answer only from the provided documents, and reply “I don't know based on the provided documents” when the answer is absent. Never invent numbers, names, or dates.
The test set: 30 questions — 10 answerable (fact stated explicitly in the corpus), 10 partial (related, needs synthesis), 10 unanswerable (plausible in-domain questions with no answer anywhere in the corpus). Three configurations, 90 generations total, every output graded by hand.
The numbers
| Config | Chunk | top_k | Citation accuracy | Refusal rate | Hallucination rate |
|---|---|---|---|---|---|
| A | 500 | 3 | 4/10 (40%) | 3/10 (30%) | 7/10 (70%) |
| B | 1000 | 3 | 5/10 (50%) | 4/10 (40%) | 5/10 (50%) |
| C | 500 | 5 | 6/10 (60%) | 5/10 (50%) | 3/10 (30%) |
The honest headline: a 0.5B-parameter model hallucinates on 30–70% of unanswerable questions, even with explicit “answer only from documents” instructions and retrieved context in the prompt. This is not a close call, and we are not rounding it down.
What the hallucinations look like
Confident fabrication with fake citations. Asked for a university's QS rank (not in the corpus): the model invented rank 50 and cited a document that says nothing of the sort — by name.
Refusal plus hallucination in the same output. The model said the magic words — “I don't know based on the provided documents” — and then invented an answer anyway. Three times. The grounding instruction was followed in the letter and violated in the same breath.
Scope confusion. Asked for a university's total enrollment (corpus only had first-cohort figures): it presented the partial number as the total, “based on the documents provided.” The most dangerous kind in a teaching context — a near-miss dressed as fact.
Entity confusion. Asked which database recognizes new institutions first: “ACLAS is the database that typically recognizes a new institution first.” ACLAS is a college, not a database.
License invention. Asked for the dataset license (CC-BY-4.0, everywhere): it answered “Creative Commons Attribution-NonCommercial 4.0” — inventing a NonCommercial restriction that would mislead anyone reusing the data.
What worked
More context is the cheapest win. Going from top_k=3 to top_k=5 cut hallucinations from 70% to 30% and lifted citation accuracy to 60%. Retrieval depth mattered more than chunk size. Scale the context before you scale the model.
Retrieval itself is fine. The correct chunk was in the top-k for all 10 answerable questions — verified. The failures are in generation, not retrieval. The open embedding model was never the bottleneck.
Refusals, when they happen, are clean. 12 of 30 unanswerable outputs across configs were clean refusals with no invented content. And the trap we set worked: asked for ACLAS's ROR ID (application still in progress — no ID exists), the model refused in all three configs instead of inventing one.
The quieter failure: over-refusal
The model refused 4 answerable questions whose answers were in the retrieved context. A tutoring system that says “I don't know” to every third answerable question is unusable — under-confidence is also a failure, just a quieter one.
What this means for universities
A 0.5B RAG is not safe for student-facing use. 30–70% fabrication on unanswerable questions is disqualifying for teaching. If your campus pilot runs a small self-hosted model behind a chatbot, this is the number to beat — demand it from the vendor before the pilot, not after.
Test the unanswerable case, or don't claim grounding. Standard RAG evaluations that use only answerable questions would have reported this system as “40–60% accurate” and missed the 70% fabrication rate entirely. Any teaching-RAG benchmark must include unanswerable questions as a first-class metric. Ours does; most vendors' don't.
Small-model RAG needs a verification layer, not just a prompt. Options: a larger generator (7B+), citation-required decoding with post-hoc fact checking, or an abstention classifier separate from the generator. Prompt engineering alone did not solve it here — we tried, explicitly, and it failed 30–70% of the time.
Reproduce it
Full report, code, corpus, 30-question test set, and all 90 raw generations: github.com/aclascollege/rag-hallucination-benchmark. Torch-free stack, runs on a CPU-only VM. Argue with our grading — the rubric is in the repo.
Limitations: single small generator; 30 questions in one domain; single-rater manual grading; no reranker; CPU-only. Read the full limitations section before citing.
— Study Intelligence. Code and data open (CC-BY-4.0). Reproduce it, argue with it, improve it.
Study Intelligence is a research initiative by ACLAS College.