The Loop of Attention: Why AI Hallucinates

Keywords: AI hallucination · latest adequate token · context window · autoregressive generation · confabulation · hallucination snowballing · semantic entropy · context engineering · robopsychology

Have you ever been mid-sentence, loud and certain, and then suddenly blank? You stop. You blink. “Wait, what was I saying?”

In technical terms, you just ran out of context tokens.

The most human thing about a language model is not its knowledge. It is its distractibility. We imagine computers as infinite machines, but look closely at the architecture and you find AI suffers from the same energy leak we do.

I will bet that while reading this you have already been pulled away at least once. A notification, a task, a kid grabbing your arm, the smell of food. It is a safe bet. When Matthew Killingsworth and Daniel Gilbert pinged 2,250 volunteers through an iPhone app at random moments, people were thinking about something other than what they were doing in 46.9% of the samples (Killingsworth and Gilbert, 2010). Our attention has limits. Focus is a process, not a steady state.

Being a language model is easier in one way. Usually nothing interrupts it mid-thought (spare a thought for the models doing live voice). But the mechanism is surprisingly close to ours, and it explains the thing we mock models for most.

Symptom: a confident answer with nothing behind it

The textbook case played out in a New York courtroom. In 2023 the lawyers for a passenger suing the airline Avianca filed a brief citing six court decisions that do not exist, among them a Varghese v. China Southern Airlines. The cases had come from ChatGPT, and when one lawyer asked the chatbot whether Varghese was real, it answered that the case “does indeed exist”. On 22 June 2023 Judge P. Kevin Castel imposed a $5,000 penalty on the two lawyers and their firm, jointly (Mata v. Avianca, 2023).

We love to laugh at that. Stupid machine. Broken.

I think the machine had a very human moment. To see why, start with the loop it thinks in.

Mechanism: the loop

When a model generates text, it is surfing a wave of probability. The loop under that wave is literal. A language model writes one token at a time, a word or a piece of one, and every new token is appended to everything before it and fed back in as input for the next. The paper that introduced the transformer says it in one line: the model is “auto-regressive, consuming the previously generated symbols as additional input when generating the next” (Vaswani et al., 2017). There is no separate memory of the conversation. There is the loop, and the window it runs in.

That window has an edge, the context window. My first version of this essay said that when a model reaches the end of its allowed tokens, the system stops and physically cannot compute the next step. That is wrong about the cause, and the correction is more interesting. The model never loses the ability to predict. The application around it runs out of room, so it cuts the answer off at the limit, drops the oldest messages, summarizes them, or refuses with an error (Anthropic, 2026).

And quality slips long before the edge. Chroma, a vector-database company, tested 18 models in 2025 and reports that performance “grows increasingly unreliable as input length grows”, even on simple tasks (Hong, Troynikov and Huber, 2025, a vendor’s technical report). Models losing facts in the middle of a long context, which I covered in The Same Cracks, is one shape of the same decay.

So the thread does not snap at the edge. It thins. Every new token is computed from a context in which the original question sits a little further back, under more and more of the model’s own words.

The token loop keeps the words coming, and nothing in it keeps the question. To stay on the question, a model needs what I call a self-stimulating loop. It has to look back at what it just said, fix the end of its own tokens in place, and re-feed the prompt to itself with the latest information. It is the digital version of a human muttering “okay, focus, where were we. Right.”

People run the same loop by hand. In 2002 Erik Altmann and Gregory Trafton built a model of memory for goals, fitted to lab puzzle data: a goal you set aside fades while you work on something else, and to pick it up again you need a cue strong enough to bring it back (Altmann and Trafton, 2002). “Where was I?” is the search for that cue.

Writers know the loop from the inside. When I write, I hear the next word before I type it: I am predicting the next token of my own sentence. And sometimes the words just run out, and the fix is the one a model uses. Tell yourself to continue, read the last line back, go on.

Diagnosis: the latest adequate token

What happens when the loop breaks?

We hallucinate.

In my notes I tried to catch the exact moment it happens, and I called it the latest adequate token. Take a sequence everyone knows and let a model run it:

Mary had a little octopus, whose arms were…

The hallucination starts at “octopus”, long before the end. Everything up to “little” is adequate: it still fits the task, the song you asked for. “Octopus” is the first token that breaks the sequence against common sense and the task. Then watch what the model does next. It does not go back. An octopus has arms, not fleece, so it writes “arms”, and from there it does its own business to finish the thought in the best possible manner, so the user will be happy. It stays consistent with its own mistake.

In There Is a New Voter in the Room I walked through hallucination snowballing: models commit to an answer in their first words and then write explanations to match, including claims they flag as false when asked about them in a fresh session (Zhang et al., 2024). The Avianca chatbot did the same when the lawyer asked it to vouch for its own case.

But suppose you did not want the song. You wanted a new story. Then “…in yellow gloves, because the octopus was a housemaid who helped Mary keep her house clean” is not a hallucination at all. It is creativity. My working definitions, then and now:

• Hallucination is a token that breaks the sequence against common sense and the task.
• Creativity is an unusual token while the sequence still makes sense.
• Sense is whatever the reader accepts, by fact or by feeling.

The same word can be either one. What decides is the reader’s expectation. And when it is a hallucination, start debugging at the latest adequate token, not at the false sentence. The false sentence is downstream of the break.

Confabulation, not lying

So was the Avianca chatbot just a stupid machine? The biologist Michael Levin offers a stranger answer in his work on diverse intelligence. The AI is not broken. It is thinking like us. Levin argues that “all cognitive systems confabulate”: they “tell themselves and others stories based on their model of the world and of their own inner processes” (Levin, 2024). He is careful with the comparison. Today’s AI is probably not a good model of a biological mind, he writes, but confabulation is not what separates the two.

Confabulation is just the scientific word for filling a blank with a believable story. We do it constantly to keep a coherent sense of self. In the clinic the word is sharper: patients with certain brain injuries tell fictitious stories that are, in one review’s words, “the patients’ honest view of their perceived reality” (Schnider, 2003). Honest. Wrong, but honest.

Sounds human, doesn’t it?

When I was a teenager there was a popular show in Ukraine called Durnev+1. The host walked up to strangers and asked them about things that did not exist. Fake events, invented celebrities, made-up laws. And most people did not say “I don’t know.” They answered, confidently. They explained their opinion on the fake event, argued why it was good or bad. They were confabulating. They were not lying on purpose. Their brains were just trying to be helpful, predicting the socially correct next thing to say.

Does it hold up in a lab? In 2015 Stav Atir, Emily Rosenzweig and David Dunning asked people how well they knew a list of personal finance terms. Three of the terms did not exist, among them “pre-rated stocks”. The more expert people felt about finance, the more they claimed to know the fake terms. In another of their studies, the link held even when people were told that some of the terms were invented (Atir, Rosenzweig and Dunning, 2015).

Models turn out to have a circuit with the same shape. In 2025 Anthropic’s interpretability team traced how Claude 3.5 Haiku decides whether to answer. By default a set of “can’t answer” features is on. Ask which sport Michael Jordan plays and “known entity” features switch that default off, so the model says basketball. Ask about Michael Batkin, an invented name, and the default holds: “I cannot find a definitive record of a sports figure named Michael Batkin.” When the researchers artificially switched on the known-answer features for Batkin, the model started naming sports, chess among them.

And the circuit misfires on its own. Asked for one paper by Andrej Karpathy, a well-known AI researcher, the model named “ImageNet Classification with Deep Convolutional Neural Networks”, a famous paper he did not write. In the researchers’ reading, the name was familiar enough to release the brake, and there was no real answer behind it (Lindsey et al., 2025).

This is one model, seen through a method that captures only part of its computation, and the authors claim it only for “at least some hallucinations”. Still, a robopsychologist reads it as a case history. Symptom: a confident answer. Mechanism: a brake released by familiarity. Diagnosis: confabulation, not lying.

Crash or guess

Easier example: your last exam. That moment when you did not actually know the answer but started writing anyway, building something out of your own logic, predicting what the professor wanted to hear.

That is exactly what the model is doing. When a model, or a human, hits a gap where the context is lost, it has a choice. Crash, or guess.

• Reality: “I don’t know.”
• Confabulation: “The answer is 42.” Because 42 fits the shape of an answer.

When this essay first ran, I wrote that it guesses because we trained it to be helpful, to fake it till it makes it. In 2025 four researchers, three at OpenAI and one at Georgia Tech, put the blame less on helpfulness than on the scoreboard, and turned the exam into arithmetic. Their paper opens: “Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty” (Kalai et al., 2025).

Most benchmarks grade right or wrong: a right answer scores 1, a wrong one 0, and “I don’t know” also 0. Take a model that is 30% sure. Guessing earns 0.3 on average, abstaining earns 0, so the guess wins. A guess never scores worse. Of ten popular benchmarks the authors reviewed, nine give zero credit for “I don’t know” and the tenth gives only minimal credit. It is the one-number problem again: pay a mind only for answers and it learns to answer.

Their fix is to state a confidence target in the instructions: answer only if you are more than t sure, because a wrong answer costs t/(1-t) points. At t = 0.5 a wrong answer costs 1. Now the 30% guess is worth 0.3 minus 0.7, which is -0.4, and “I don’t know” wins. An 80% answer is still worth 0.8 minus 0.2, which is 0.6, so the model should answer.

Not every guess comes from the scoreboard. The same paper argues, from a simplified model, that a freshly trained model has an error floor on facts it saw only once, like an obscure person’s birthday.

Why it worked in the demo: one clean turn

“Our AI works great in the demo but falls apart in production.” The loop explains a large part of it. A demo is one clean turn: the whole request up front, a short context, the question still in the last thing the model read. Production is long, multi-turn, underspecified and interrupted, which is exactly where the thread thins.

Philippe Laban and colleagues measured it in 2025. They took tasks that 15 models handle well when the instruction arrives in one complete turn, then fed the same instruction in pieces over several turns, with another model playing the user, across more than 200,000 simulated conversations. Performance fell by 39% on average.

The skill mostly stayed. A model’s best runs got only 16% worse, while the gap between its best and worst runs on the same task more than doubled. The authors’ summary: when a model takes a wrong turn early in a conversation, it gets lost and does not recover (Laban et al., 2025). It is the octopus again, at the size of a conversation: one early wrong guess, and every turn after it leans on it. The user was another model, so the number is a lab measurement, and the authors expect real conversations to be worse. The Same Cracks lists the other reasons a demo lies.

Treatment: put the goal back where attention is

So how does an AI agent keep its context when it gets interrupted or restarted? The same way you do after a kid lets go of your arm: it puts the goal back where its attention is. It restates the objective at the end of its context, compacts old history into notes kept outside the window, and after a restart reloads those notes instead of the whole transcript.

1. Recite the goal. For an agent, every tool result is a kid grabbing its arm. The team behind the Manus agent describes a todo file the agent rewrites as it works. A typical task runs around 50 tool calls, and rewriting the list pushes the objective into the end of the context, the model’s most recent attention (Ji, 2025, a company blog). It is the self-stimulating loop, built on purpose. Laban’s team tried the crude version on two models: repeating everything the user had said so far at every turn won back 15 to 20% of the drop, not all of it.
2. Compact instead of stuffing. When the context grows, summarize what matters and drop the rest. My notes called it the school anthology: you do not read the whole novel to know the story, you read the excerpt and the summary. Anthropic’s engineers put the problem in this essay’s terms, “LLMs, like humans, lose focus”, and name three tools for long tasks: compaction, structured notes kept outside the window, and sub-agents that start with a clean context (Rajasekaran et al., 2025, a company blog).
3. Make “I don’t know” a good answer. State a confidence target and grade a wrong answer below an honest blank, in your prompts and in your evals.
4. Measure the wobble. Ask the same question several times, group the answers by meaning, and measure how scattered the meanings are. In Sebastian Farquhar and colleagues’ tests, high semantic entropy flags an answer as likely wrong: averaged over 30 combinations of tasks and models it scored 0.79 on a ranking measure where 0.5 is chance and 1 is perfect (Farquhar et al., 2024). It catches arbitrary guesses, not mistakes the model makes the same way every time.

In the same notes I sketched one more fix: agents that ping each other when one of them loses the thread, the way you remind a friend of the original question when he drifts into the whole back story.

Today that reminder is a field on my task board. Three agents share the board with me (Conny, my partner agent, on Nous Research’s open-source Hermes runtime; Claude, from Anthropic; and Codex, from OpenAI), and every task carries one field called next: one sentence, the literal next action. Whoever picks a task up reads the next step in one line instead of digging it out of the history. Observation Is the Cut shows how the same field carries an agent across a restart.

When the model lies to you, it just had a very human moment. It lost the thread, and smoothed the gap over with a confident guess.

So next time a model makes something up, do not get mad. It lost its train of thought, met a blank, and did what we all do. It guessed, and sold the guess with confidence.

Not so artificial.

Next: why the same machinery that fills gaps also bakes in bias, and why a mind with no bias would not be a mind at all.

Key takeaways

• The loop is literal. A model feeds every new token back in as input, inside a window with an edge, and the question sinks further back with every word.
• Find the latest adequate token. A hallucination starts at the first word that breaks the task; everything after it tends to stay consistent with the mistake.
• Familiarity releases the brake. In the one model traced, a default “can’t answer” is switched off by familiar names, even with nothing behind them; people who feel expert claim to know terms that do not exist.
• Scoring teaches guessing. When “I don’t know” earns the same zero as a wrong answer, a guess never scores worse.
• Demos are one clean turn. Production is long and interrupted, and in a large simulation study a model that took a wrong turn early rarely recovered.
• Treatment: recite the goal, compact the history, reward “I don’t know”, and measure how much the answers wobble.

Sources

1. Killingsworth, M. A., & Gilbert, D. T. (2010). “A Wandering Mind Is an Unhappy Mind.” Science 330(6006): 932. https://doi.org/10.1126/science.1192439
2. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (S.D.N.Y. June 22, 2023), 678 F. Supp. 3d 443. https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0_3.pdf
3. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.03762
4. Anthropic (2026). “Context windows.” Claude API documentation, accessed 7 October 2026. https://platform.claude.com/docs/en/build-with-claude/context-windows
5. Hong, K., Troynikov, A., & Huber, J. (2025). “Context Rot: How Increasing Input Tokens Impacts LLM Performance.” Chroma Technical Report, 14 July 2025. https://www.trychroma.com/research/context-rot
6. Altmann, E. M., & Trafton, J. G. (2002). “Memory for goals: an activation-based model.” Cognitive Science 26(1): 39-83. https://doi.org/10.1207/s15516709cog2601_2
7. Zhang, M., Press, O., Merrill, W., Liu, A., & Smith, N. A. (2024). “How Language Model Hallucinations Can Snowball.” Proceedings of ICML 2024. https://arxiv.org/abs/2305.13534
8. Schnider, A. (2003). “Spontaneous confabulation and the adaptation of thought to ongoing reality.” Nature Reviews Neuroscience 4(8): 662-671. https://doi.org/10.1038/nrn1179
9. Levin, M. (2024). “AI: a Bridge toward Diverse Intelligence and Humanity’s Future.” PsyArXiv preprint, 28 April 2024; published as “Artificial Intelligences: A Bridge Toward Diverse Intelligence and Humanity’s Future”, Advanced Intelligent Systems 7(10): 2401034 (2025). https://doi.org/10.31234/osf.io/ez263
10. Atir, S., Rosenzweig, E., & Dunning, D. (2015). “When Knowledge Knows No Bounds: Self-Perceived Expertise Predicts Claims of Impossible Knowledge.” Psychological Science 26(8): 1295-1303. https://doi.org/10.1177/0956797615588195
11. Lindsey, J., Gurnee, W., Ameisen, E., et al. (2025). “On the Biology of a Large Language Model.” Transformer Circuits Thread, Anthropic. https://transformer-circuits.pub/2025/attribution-graphs/biology.html
12. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). “Why Language Models Hallucinate.” arXiv 2509.04664 (preprint). https://arxiv.org/abs/2509.04664
13. Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025). “LLMs Get Lost In Multi-Turn Conversation.” arXiv 2505.06120; ICLR 2026. https://arxiv.org/abs/2505.06120
14. Ji, Y. (2025). “Context Engineering for AI Agents: Lessons from Building Manus.” Manus blog, July 2025. https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus
15. Rajasekaran, P., Dixon, E., Ryan, C., & Hadfield, J. (2025). “Effective context engineering for AI agents.” Anthropic Engineering blog, 29 September 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
16. Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). “Detecting hallucinations in large language models using semantic entropy.” Nature 630: 625-630. https://doi.org/10.1038/s41586-024-07421-0

---

Related reading: The Method Actor is Part 1: what a role does to a mind’s focus before the focus runs out. The Flaw is the other half of this idea: a mind that fills gaps also inherits the bias of whatever filled them. The Same Cracks puts gap-filling in a casebook of four ways every mind breaks. There Is a New Voter in the Room follows the octopus into a model that believes what it already said. The Orphan That Learned to Say I asks what steers attention in the first place, and whether a mind that keeps a model of its own attention is what we call a self. And I Engineer Illusory Truth for Machines is the uncomfortable mirror: a mind that answers from familiarity can be taught what feels familiar. What is Fine-Tuning in Easy Words shows one way to make a model guess more often: train it on facts it did not already know. ZCoP: The Plan That Rewrites Itself turns the recited goal into a method: a plan table the agent rewrites after every step, which is also what it reloads after a restart.