# Long Context Hallucinations: Why the Demo Is Not the System Source: https://normanholz.de/articles/long-context-hallucination-demo/ Language: en Last updated: 2026-05-29 Reading time: 9 min read LLMs can pass a small document demo and still fail when the same task scales. A simple invoice benchmark shows why deterministic workflows still need deterministic software. Many clients ask why I push back when someone wants to put an LLM or an agent on every slow internal process. The argument usually sounds reasonable at first. A person can do the task. The steps are known. The documents are available. The answer is just a matter of reading, extracting, checking, and adding. A chatbot handled three sample files in a demo. Why not automate the whole thing? Because the demo is not the system. Traditional software works differently. If a person can describe the process end to end, and if every rule and calculation can be specified, software can turn that process into deterministic behavior. The same input produces the same output. Edge cases still exist, but they can be named, tested, and fixed. LLMs do not give you that kind of system by default. They generate the next token from probability. You can lower the temperature. You can improve the prompt. You can use structured output. You can add retrieval and validation. All of that helps. None of it turns a probabilistic model into a deterministic program. That difference matters most when the task looks boring. ## Small demos create false confidence Most AI demos are too small to reveal the failure mode. Upload three invoices. Ask for the total. The model answers correctly. Upload five contracts. Ask for a summary. The model finds the obvious clause. Upload a few policy documents. Ask a question. The answer looks useful. For a non-technical stakeholder, this can feel like proof. The tool handled the task. The next step seems obvious: connect the API, upload all documents, and let the model do the same thing at scale. That is where the assumption breaks. A model that works on a small context can fail in a large one. Not because the task became intellectually harder. The task may still be simple. The problem is that the model has to keep more facts, more instructions, more exceptions, and more distractors in its active context. The larger the prompt, the more room there is for skipping a document, using the wrong number, losing the extraction rule, or making an arithmetic mistake. This is why I am careful with phrases like "the model supports one million tokens." Theoretical context size is not the same thing as reliable working context. ## Long context does not mean reliable context There is useful research on this. [Lost in the Middle](https://arxiv.org/abs/2307.03172) showed that models can perform worse when the relevant information sits in the middle of a long input, even when that information is present. [Context Is What You Need](https://arxiv.org/abs/2509.21361) uses the term "maximum effective context window" for a reason. The question is not only how many tokens a model can accept. The question is how much of that context it can use effectively for a real task. [Shadows in the Attention](https://arxiv.org/abs/2505.16894) looks at how context perturbation and representation drift relate to hallucinations. The details are technical, but the practical point is simple enough: adding context can change model behavior in ways that are not obvious from the prompt alone. This also explains why many proprietary chat products limit uploads, file counts, or practical context use even when the underlying model advertises a much larger window. Some of that is product design, cost, latency, abuse prevention, and user experience. Reliability is part of the same picture. A large context window is useful. It is not a guarantee. ## The invoice test To make this easier to explain to non-technical teams, I built a small controlled experiment. The task is deliberately boring. Generate a large number of artificial documents. Ask a model to read them all. Extract the relevant integer values. Return the sum. That is it. The point is not that addition is hard. The point is that the correct answer is known exactly. If the model gets it wrong, we do not need a debate about taste, judgment, domain expertise, or writing style. The model failed a deterministic extraction and summing task. The benchmark has three modes. ## 1. Fixed invoice totals The first mode creates invoices where every file contains the same target line: ```text Total amount due: 10 EUR ``` If there are 100 documents, the answer is 1,000. If there are 1,500 documents, the answer is 15,000. This is intentionally simple. It tests whether the model can keep the document count and extraction rule straight across a growing context. ## 2. Index values The second mode creates documents with a value that matches the document index: ```text Value: 37 ``` The ground truth is the familiar formula: ```text num_docs * (num_docs + 1) / 2 ``` This removes trust from the model. The benchmark knows the answer without asking the model to calculate it. ## 3. Approved random invoices The third mode is closer to real work. It creates invoice documents with seeded totals, irrelevant distractor numbers, and a status line. Some invoices are approved. Some are void. Only approved invoices count. The model sees documents with values such as invoice numbers, account numbers, purchase order references, prior balances, approval codes, and target totals. It has to follow the rule: ```text Only sum Total amount due values where Status is approved. ``` The totals look random enough that the model cannot solve the task by spotting a simple pattern. The benchmark still knows the answer because the documents are generated from a fixed seed. That is the useful part. The task is easy to explain, but unforgiving. The model either extracts the right values and sums them, or it does not. ## Why temperature 0 is not a magic fix The benchmark runs each case at two temperatures: - `0.0`, which asks the model to behave as consistently and carefully as possible - `0.7`, which is closer to the sampling style often used in chatbots and agents This comparison matters. If a model fails at temperature `0.7`, someone can say the sampling was too loose. Fair enough. If it fails at temperature `0.0`, the problem is harder to dismiss. The failure is less likely to be random style variation and more likely to come from context handling, extraction, arithmetic, output formatting, or the model's internal handling of the task. Temperature 0 reduces randomness. It does not make the model a calculator, parser, database, or workflow engine. ## Separate hallucinations from operational failures The suite separates two kinds of problems. Wrong completed sums count as hallucinations. The model returned an answer, but the answer disagreed with deterministic ground truth. Timeouts, API errors, context-length failures, and invalid structured output are reported separately as operational failures. That distinction matters in real projects. If the model returns a wrong sum, you have a correctness problem. If the model times out or cannot produce valid JSON, you have a reliability problem. Both can break the workflow. They just break it in different ways. For local Ollama models, large prompts may hit runtime context limits before you see a hallucination. That is not a win. It means the local runtime could not operate at the size the workflow required. ## How to run the experiment The repository is a small `uv` project. The full suite generates all three experiment types for 10, 100, 500, 1000, and 1500 documents. It repeats each case and writes an integrated report. A typical local run looks like this: ```bash uv run -m long_context_demo.cli suite run \ --ollama-model ollama:gpt-oss:20b \ --ollama-model ollama:gemma2:9b \ --timeout-seconds 600 \ --output-dir reports ``` The output includes: ```text reports//results.jsonl reports//summary.md reports//config.yaml reports//result_table.svg ``` The table is useful for articles, stakeholder discussions, and internal architecture reviews. It makes the failure mode visible without asking anyone to read transformer papers first.
Result table for a Gemma 2 9B long-context summing benchmark, showing many wrong completed sums across document counts and temperatures.
Example run with Gemma 2 9B: 90 completed calls, 72 wrong completed sums, and no operational failures.
## What this proves, and what it does not prove This experiment does not prove that LLMs are useless for document workflows. They are useful. They can classify, summarize, extract, normalize, draft, search, and assist. In many workflows they save real time. The experiment proves something narrower: a model can fail on a simple deterministic task when the context grows, even when all required information is present in the prompt. That is enough to change the architecture discussion. If the answer must be exact, do not ask the model to be the source of truth. Use deterministic code for deterministic parts: - Parse structured data with parsers - Store extracted values in a database - Use schemas and validation - Calculate totals in code - Keep audit logs - Compare model output against known constraints - Put humans in the loop where judgment matters Use the LLM where it is strong: - Turning messy text into candidate structured data - Explaining uncertain cases - Ranking documents for review - Drafting summaries for humans - Handling natural language input around a deterministic core That is a different architecture from "send all documents to an agent and trust the answer." It is less magical. It is also how production systems survive contact with real data. ## The decision rule When a process is time consuming, the first question should not be: > Can an LLM do this? The better question is: > Which part of this process requires language judgment, and which part requires deterministic correctness? Those are different jobs. If a human can explain every calculation and every rule, write software for that part. Use tests. Use known inputs. Use known outputs. Make the behavior boring. If the input is messy language, use the model to help turn it into something the software can check. Do not confuse those two layers. The model can assist the workflow. It should not silently become the accounting system, compliance engine, or final decision maker just because the demo looked good. Small demos answer one question: can this work once? Production architecture has to answer a harder one: can this fail safely when the context grows, the documents get messy, and the answer matters? That is the question worth designing for.