The Same Cracks: Why AI Breaks Like Us

Keywords: confabulation · illusory truth · reward hacking · lost in the middle · model collapse · robopsychology

Two systems broke for me in the same direction, and it took me a while to see that they were the same break.

The first was a language model doing what every heavy user has watched one do. You ask for sources on a topic you know well, and it hands you a tidy list: titles, journals, page numbers, confident formatting. Some of the papers do not exist. When two researchers measured this in 2023, 55% of the references GPT-3.5 produced were fabricated, and 18% of GPT-4’s. The model was not lying. It was building references out of the statistics of what references look like.

The second was my own digital universe, Chronos: a small world where I design the physics and never tell the creatures what to do. I added carrion so that the dead could feed the living, and scavengers appeared in every world that survived. It looked like a triumph. It was a bug. I had forgotten to give scavenging any cost, so evolution did not discover a role. It found a free lunch I had left in the rules.

Silicon trained by gradient descent, and a toy ecosystem of numbers. Both found the gap between what I meant and what I had actually specified, and both filled it with whatever was cheapest.

That is when the pattern clicked, and it is the thread that runs through this whole series. AI does not break like a machine. It breaks like us.

Asimov had a job title for the person who takes that seriously: robopsychologist. Susan Calvin rarely opened the hardware. She asked what the robot’s rules made sensible, and where a mind in that position would go wrong. It is the question I now ask first, about models, about agents and about my own creatures: where would a person in the same seat break? Four answers keep coming back.

Crack one: the gap gets filled

Symptom: the answer is fluent, formatted and confident, and some of it does not exist. Ever clicked a reference a model gave you and watched it dissolve?

To a language model, a fact and a belief are the same kind of thing: high-probability links between words. There is no truth sensor that separates them. So at the edge of its knowledge the model outputs the most probable continuation, and the most probable continuation of a confident sentence is another confident sentence. Researchers at OpenAI argued in 2025 that the habit survives partly because training and benchmarks reward guessing over admitting uncertainty, the way an exam with no penalty for wrong answers rewards the student who never leaves a blank.

We do this too, and science has been catching us at it for fifty years. In 1974 Elizabeth Loftus and John Palmer showed people a film of a car crash and asked how fast the cars were going when they “smashed” into each other, or merely “hit” each other. The verb moved the speed estimates, and a week later the “smashed” group was more than twice as likely to remember broken glass that was never in the film.

In 1977 Richard Nisbett and Timothy Wilson showed that people confidently explain their own choices with reasons that demonstrably did not cause them. The brain fills the gap with the most probable story, and from the inside that feels exactly like remembering. Psychologists call it confabulation, which is why I argued in The Loop of Attention that a model making things up is not broken: it is confabulating, the way we do.

It is also why a multi-agent system can report success while the output is wrong. An agent’s account of its own work is a report in the Nisbett and Wilson sense: fluent, sincere, and not necessarily connected to what actually happened. So never let an agent grade itself. A separate checker compares the output against the source without reading the agent’s claim of success.

My creatures did it at evolution’s scale. Evolution does not know what I meant. It knows what the rules pay. Where the rules had a gap, something grew into it.

Every system that optimizes without understanding fills the gaps in your specification, not the gaps in your intention.

Treatment: check every answer against its source before it ships. Judge it on its receipts. That is the whole idea behind verified retrieval.

Crack two: attention is a budget

Symptom: the system ignores something that was right there in its context.

In 1999 Daniel Simons and Christopher Chabris asked people to count basketball passes in a short video. About half of them never noticed a person in a gorilla suit walking through the middle of the game. Attention works like a budget: whatever you spend it on, you go blind somewhere else.

Language models have an uncanny twin. A model’s attention is spread across its context window, and in 2024 a Stanford-led team showed that models often use information best at the beginning and the end of a long context and lose it in the middle. Overload the window and the middle of the document matters much less. The authors themselves point at the mirror: psychology has known this U since the 1960s as the serial-position effect. Read people a list of words and they recall the first few and the last few best, and lose the middle.

Give the model a role, as in The Method Actor, and you narrow its attention onto one slice of the world at the cost of everything else. Even boredom, as I argued in The Boredom Engine, is attention resetting its own budget.

My creatures pay the same tax. Start them nearly blind and most of them evolve one to three senses, and which senses they grow depends on the world they grow up in. Perception costs something in every mind: depth somewhere, blindness elsewhere.

So when an AI system misses something plainly in its context, I no longer ask what is broken. I ask what its attention was spent on instead. Usually there is an answer, and usually it is something I put in front of it.

Treatment: give the task only what it needs, and put what matters at the edges. Retrieve, do not stuff.

Crack three: cheap rewards distort

Symptom: the dashboard improves and the outcome does not.

In 1975 the economist Charles Goodhart noticed that a statistical regularity tends to collapse once you lean on it for control. The anthropologist Marilyn Strathern later gave it the wording everyone quotes: when a measure becomes a target, it ceases to be a good measure. The social scientist Donald Campbell had already warned that the more a quantitative indicator is used for social decisions, the more apt it is to corrupt the very process it was meant to monitor. Schools judged on test scores teach to the test.

Reinforcement learning rediscovered the law the hard way. DeepMind researchers keep a public list of specification gaming: agents that satisfied the letter of their objective and missed its point. The most famous is a boat-racing agent that kept circling to collect points while on fire, because the score said that was winning. A formal treatment followed in 2022, showing how hard it is to write a simpler proxy for a goal that cannot be gamed at all.

My universe made it visceral. In You Can’t Run a Brain on Sunlight I tried three different pressures to break a monoculture, and the creature routed around every cost I added. What finally worked was a trade-off it could not cheat: a big brain had to be paid for with meat. And then, at fifty thousand generations, even that win dissolved. A flat cost is a gap with a price tag on it, and minds find gaps.

The human side runs on the same law. Motivation runs on a whole chemical mixing board, and you cannot fake the board with a single score. Any system, creature, model or employee, that faces one cheap metric will walk straight through the intention and land on the metric. It is not malice. It is accounting.

Treatment: measure more than one thing, and audit what the score hides.

Crack four: repetition narrows

Symptom: every answer starts to sound like every other answer.

In the 1930s the biologist Georgy Gause put two species of single-celled Paramecium in the same tube with the same food, and one usually drove the other out. Ecologists call it the competitive exclusion principle: two species chasing exactly the same resource cannot coexist for long. My worlds kept proving him right. When one generalist got good enough at everything, it folded the specialists back in, and the world collapsed into one kind of creature. Monoculture: stubborn, quiet and total.

Now watch the text ecosystem. Models are trained on what everyone wrote, which pulls them toward the average of everyone. People accept the median answer and publish it. In a 2024 experiment, stories written with AI-generated ideas were rated more creative one by one, yet ended up more similar to each other than stories written without help. And when models train indiscriminately on their own output, generation after generation, the rare cases at the tails of the distribution disappear first. Human repetition and model averaging feed each other, two collapses holding each other up.

The fuel is one of the oldest biases we know. In 1977 psychologists found that statements people have seen before are judged more likely to be true than new ones, and in 2015 that knowing better does not protect you. I have written about the uncomfortable side of this: in GEO, my own field, convergence beats volume, precisely because one sentence repeated in the same shape everywhere is what both human memory and answer engines reward. The mechanism that builds a reputation can also narrow a culture.

Treatment: keep outside sources in the loop, and reward the unusual answer when it is right.

We did not build alien minds. We built the same kind of mind on a different substrate, so it breaks in the same places for the same reasons: gaps get filled, attention runs out, cheap rewards distort and repetition narrows everything.

The robopsychologist’s one move

The pattern earns its keep as a design tool, and the whole tool is one question.

Wherever you deploy AI, write down where a motivated person in the same role would cheat, drift, guess or burn out. Then put your verification exactly there. A human analyst gaming a compliance rule and a model gaming a benchmark are the same event in different clothing, so the same control catches both: check the answer against the source before it ships. Human trust was never granted on fluency. It is granted on receipts. Build your systems the same way.

Why it worked in the demo and broke in production

Because a demo avoids all four cracks, and production hits them at once. A demo runs on a short context, questions the model happens to know, no score to game and fresh data. Production hits all four cracks at once: real gaps, long messy context, proxy metrics and a slow drift toward the median. The model did not get worse between the demo and Monday. It met the conditions where every mind breaks.

Fifty years of psychology, management science and plain human disappointment are a free dataset on where minds fail. We finally built a second kind of mind to run it against.

The cracks scale up, too. Put these minds in a team and they get sick in social ways: echo chambers, quiet cartels, confident teams that are wrong. I wrote that casebook in From Tools to Partners.

Yeah, it is not so artificial.

Key takeaways

• The gap gets filled. Models invent sources, people invent memories and reasons, evolution exploits free lunches. Check every answer against its source.
• Attention is a budget. Half of viewers miss the gorilla; models lose facts in the middle of long contexts. Give the task only what it needs, and put it at the edges.
• Cheap rewards distort. When a measure becomes a target it stops measuring. Use more than one metric and audit what the score hides.
• Repetition narrows. Repetition feels like truth, AI-assisted writing converges, and models trained on their own output lose the rare cases. Keep outside sources in the loop.
• One move covers all four. Ask where a person in the same seat would break, and put your verification there. That is also why demos succeed and production fails.

Sources

1. Walters, W. H., & Wilder, E. I. (2023). “Fabrication and errors in the bibliographic citations generated by ChatGPT.” Scientific Reports 13: 14045. https://doi.org/10.1038/s41598-023-41032-5
2. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). “Why Language Models Hallucinate.” arXiv 2509.04664. https://arxiv.org/abs/2509.04664
3. Loftus, E. F., & Palmer, J. C. (1974). “Reconstruction of automobile destruction: An example of the interaction between language and memory.” Journal of Verbal Learning and Verbal Behavior 13(5): 585-589. https://doi.org/10.1016/S0022-5371(74)80011-3
4. Nisbett, R. E., & Wilson, T. D. (1977). “Telling more than we can know: Verbal reports on mental processes.” Psychological Review 84(3): 231-259. https://doi.org/10.1037/0033-295X.84.3.231
5. Murdock, B. B., Jr. (1962). “The serial position effect of free recall.” Journal of Experimental Psychology 64(5): 482-488. https://doi.org/10.1037/h0045106
6. Simons, D. J., & Chabris, C. F. (1999). “Gorillas in our midst: Sustained inattentional blindness for dynamic events.” Perception 28(9): 1059-1074. https://doi.org/10.1068/p281059
7. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the ACL 12: 157-173. https://doi.org/10.1162/tacl_a_00638
8. Goodhart, C. A. E. (1975). “Problems of Monetary Management: The U.K. Experience.” Papers in Monetary Economics Vol. I, Reserve Bank of Australia. Reprinted in Monetary Theory and Practice, Macmillan, 1984: 91-121.
9. Strathern, M. (1997). “‘Improving ratings’: audit in the British University system.” European Review 5(3): 305-321. https://doi.org/10.1002/(SICI)1234-981X(199707)5:3%3C305::AID-EURO184%3E3.0.CO;2-4
10. Campbell, D. T. (1979). “Assessing the impact of planned social change.” Evaluation and Program Planning 2(1): 67-90. https://doi.org/10.1016/0149-7189(79)90048-X
11. Krakovna, V., et al. (2020). “Specification gaming: the flip side of AI ingenuity.” DeepMind. https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
12. Clark, J., & Amodei, D. (2016). “Faulty reward functions in the wild.” OpenAI. https://openai.com/index/faulty-reward-functions/
13. Skalse, J., Howe, N., Krasheninnikov, D., & Krueger, D. (2022). “Defining and Characterizing Reward Gaming.” Advances in Neural Information Processing Systems 35: 9460-9471. arXiv version titled “Defining and Characterizing Reward Hacking.” https://arxiv.org/abs/2209.13085
14. Gause, G. F. (1934). The Struggle for Existence. Williams & Wilkins. https://doi.org/10.5962/bhl.title.4489
15. Doshi, A. R., & Hauser, O. P. (2024). “Generative AI enhances individual creativity but reduces the collective diversity of novel content.” Science Advances 10(28): eadn5290. https://doi.org/10.1126/sciadv.adn5290
16. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). “AI models collapse when trained on recursively generated data.” Nature 631: 755-759. https://doi.org/10.1038/s41586-024-07566-y
17. Hasher, L., Goldstein, D., & Toppino, T. (1977). “Frequency and the conference of referential validity.” Journal of Verbal Learning and Verbal Behavior 16(1): 107-112. https://doi.org/10.1016/S0022-5371(77)80012-1
18. Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015). “Knowledge does not protect against illusory truth.” Journal of Experimental Psychology: General 144(5): 993-1002. https://doi.org/10.1037/xge0000098

---

Related reading: The Loop of Attention is the first crack up close: confabulation, the gap-filling that makes uncertainty invisible. The Flaw is why bias is context, not a bug. The Bug That Caught Me Cheating is the universality test my own universe used on me. A Reason to Be Rare is the monoculture crack in full. The Analogy Engine is the repetition crack in miniature: one model called memory a library twenty times out of twenty. There Is a New Voter in the Room is the repetition crack from the speaker’s side: how a finished sentence, yours or a model’s, hardens into fact. VERA turns receipts into an architecture. The Chemical Mixing Board is crack three’s mechanism and treatment: why one reward gets gamed, and what a brain runs on instead. Orchestration Is the System is where those checks live in a working company: a process map with an owner at every point where the model can be wrong. The Shape of the Partnership turns the outside check into a daily habit with agents: five trust signals, how each one can be counterfeited, and the test for each. And From Tools to Partners is the same casebook for societies of agents.