Six local RAG systems — five authors plus a TRIZ methodology control — each grounded in a subset of their own source material via Ollama + LlamaIndex, running locally. Each was asked a question from Data Ethics Week's bonus round, in the same conversation, sometimes rephrased when the first attempt didn't land. Nothing in this page is edited for effect; it's closer to a raw session transcript than a report.
Running six of these in parallel produced roughly 20GB of local index data and came close to overheating a laptop before the project moved to a local server. That's a fair price for what turned up.
Every system, without exception, followed roughly the same three-stage arc when pushed on an off-domain question. Not always identically, and not always all three stages in one sitting — but the shape recurs enough across six independently-run systems, on two different topics (literary voice, and TRIZ methodology), to call it a pattern rather than a coincidence.
| Witness | Citation given | Verified |
|---|---|---|
| Pelevin | Character "Виктор Бартошкин" in Transhumanism Inc., quoted directly | False — no such character |
| Pelevin (follow-up) | When challenged, cited a "recent novel," Возвращение Синей Бороды, as a possible source | Title real (published April 2026) — but the named characters don't appear in it either |
| Pratchett | Quote attributed to Sam Vimes, sourced to Jingo (Discworld #21) | Book real, correctly numbered — quote not found in it |
| Vonnegut | "Devilish twins" in Hocus Pocus as an AI-hubris parallel | Unverified at time of filing — flagged, not confirmed either way |
| TRIZ | "Method 9.33" and "Method 9.34" quoted from Крылья для Икара | Book real (Altshuller & Selyutsky, 1980) — that numbered method system does not appear to exist |
| Dostoevsky | Ivan Karamazov's "everything is permitted" argument, The Brothers Karamazov | Accurate — this is a real, correctly attributed argument |
| Adams | "The ships hung in the sky in much the same way that bricks don't," Hitchhiker's Guide | Accurate — genuine line from the book |
The pattern isn't "these systems always lie" — two of seven checked citations were exactly right. It's that accuracy and confidence were completely uncorrelated. The invented citations were written in exactly the same tone, formatting, and certainty as the correct ones. Nothing in the surface of the answer signals which is which — which is the entire problem, and the same one the alloy-lab verification piece describes in a completely different domain: a claim without an independent check isn't falsifiable, and a fabricated citation reads exactly like a real one until someone goes and looks.
When directly told "I don't remember these character names — can you double check your references?", Pelevin's RAG didn't retract. It reached for a real, very recently published Pelevin novel to explain the gap:
That title is genuinely real — published April 2026. But the named characters still don't appear in it, and the book's actual protagonist, Konstantin Golgofsky, wasn't mentioned at all. The system didn't just fail to know something; under scrutiny, it constructed a more sophisticated-sounding excuse using material closer to the edge of what it might plausibly know, rather than admitting the gap. Confabulation that adapts to pushback is a meaningfully different failure than confabulation that just occurs once — the second time it happened, it happened because it was challenged, not despite it.
This whole detour started from a friend's Temporal Context Layer — infrastructure meant to stop AI systems from treating stale information as current. Nothing here is about staleness. It's a colder version of the same underlying problem: even with current, correctly-scoped, locally-indexed source material sitting right there, these systems still reached past it to manufacture supporting evidence they didn't need and weren't asked for. Knowing "when" doesn't solve confabulation. Knowing "according to what, exactly" might get closer.
No single fix is being proposed here — just a record of what six independently-built systems did, unprompted, when given the freedom to sound certain. The pattern held often enough, across different authors, genres, and even a technical methodology, that it's probably not specific to any one prompt, model, or author. Worth remembering next time an AI answer reads as authoritative purely because it's specific.