The Flaw: Why You Cannot Code Bias Out of AI
Keywords: AI bias · inductive bias · likelihood vs truth · word embeddings · WEAT · debiasing · covert bias · dialect prejudice · illusory truth · context · foreign-language effect · robopsychology
We want AI to be objective. We shout at developers to ship unbiased models. But dig into how context actually works and you hit an uncomfortable truth. Intelligence requires bias.
Let me explain with a sitcom. Remember the doppelgängers from How I Met Your Mother? For a whole season the characters keep spotting bizarro versions of themselves. Lesbian Robin. Mustache Marshall. Stripper Lily. The hunt ends with the gang’s own diagnosis: you see what you want to see.
Now look at me. My name is Dima. I am blond. Ideally, when you meet me, you see “just a guy.”
But imagine a man, call him Ted, with a very specific set of training data. Maybe as a kid a blond named Dima stole his lunch money. Maybe an ex ran off with a blond Dima. Maybe the villain in a film he loved was a blond Dima.
In Ted’s network, the probability of “blond plus Dima equals dangerous” is sky high. So when I walk up and offer him a coffee, what does his brain predict? Poison.
It does not matter that I am kind. It does not matter that I am smiling. His context overrides the reality in front of him.

Ted’s brain did what brains are for: it learned from what it saw and predicted from what it learned. The trouble is what it saw. In The Loop of Attention this machinery filled a gap in a conversation with a confident guess. Here the gap is a stranger, and the guess is poison. Hold on to Ted. He is the whole case.
Symptom: the mirror learned our associations
In 2017 Aylin Caliskan, Joanna Bryson and Arvind Narayanan took a classic psychology test, the Implicit Association Test, which times how fast people pair concepts such as flowers or insects with pleasant or unpleasant words. They rebuilt it for word vectors, the coordinates a model gives each word in a space of meaning, here learned from a web crawl of about 840 billion tokens, roughly words.
Every implicit bias they tested came back. Flowers sat closer to pleasant words than insects did. European-American names sat closer to pleasant words than African-American names. Male names sat closer to career, female names to family. And across 50 occupations, how strongly a job title leaned female tracked the real share of women in that job, at a correlation of 0.90 (Caliskan et al., 2017).
Nobody typed “nurse is female” into those vectors. The web said it over and over, quietly, in the company the words keep. Ted’s lunch money at the scale of a language. One limit: this was one set of word vectors, and the test measures associations in text, not what anyone does with them. In The Measure of Misunderstanding I described every person as running a private embedding model trained on their own life. This one is public, and trained on all of ours.
Mechanism: a likelihood detector
And now the kicker. To a language model, facts and beliefs look much the same.
- The fact: the Earth orbits the Sun.
- The belief: blond Dimas are dangerous.
Both are just high-probability links between vectors. The model is trained to predict what is most likely to come next given its history, not what is objectively true.
When I first published this essay I wrote that the model has no truth detector, only a likelihood detector. The second half holds. The first half was too strong, and the correction makes the case stranger.
Researchers have looked inside. Samuel Marks and Max Tegmark found that the larger LLaMA-2 models represent whether a simple statement is true (“The city of Tokyo is in Japan” against a false version) along a single direction in their internal activity, and shifting activity along that direction could make a model treat false statements as true, and true ones as false (Marks and Tegmark, 2024).
Collin Burns and colleagues found a direction like it in six older models with no labels at all, and when a misleading prompt dragged one model’s answers from 80% right to 71%, the direction stayed right 84% of the time. They called it a first step toward “discovering what language models know, distinct from what they say” (Burns et al., 2023).
Amos Azaria and Tom Mitchell trained a small classifier on the hidden layers of two small open models and predicted whether a statement was true 71% to 83% of the time (Azaria and Mitchell, 2023).
So there is a whisper of truth in there. The output does not have to obey it. TruthfulQA, a test of 817 questions that some people answer wrongly because of a popular misconception, found the best model of 2021 truthful on 58% of them against 94% for people. Among the models tested, the largest were generally the least truthful: better at saying what people often say (Lin, Hilton and Evans, 2022). Those were models without training from human feedback, so the lesson is about imitation.
Kenneth Li and colleagues then nudged a model’s activity along truth-related directions while it answered, and its score for answers both true and informative rose from 32.5% to 65.1% (Li et al., 2023). They found the directions with part of the same test, and on other question sets the gain was small.
A limit before anyone builds a lie detector on this: the early probes were trained on simple, curated statements, and they failed to generalize in basic ways, to negated sentences among them. The title of that paper says it: still no lie detector for language models (Levinstein and Herrmann, 2024).

So I would write the line differently today. The model has a likelihood detector, and under it a weak sense of what is true that the likelihood is free to outvote. Ted can see that I am smiling. His prediction still says poison.
Why propaganda works
This is why propaganda is so terrifyingly effective on humans. When a state screams the same lie from every channel, every poster, every radio, what is it doing? It is re-weighting your context window. It is artificially inflating the probability of that lie until your brain, hungry for efficiency and comfort, predicts it as truth.
Psychology measured this half a century ago: statements people have seen before get rated as more likely true than new ones, and knowing the right answer does not reliably protect you (Hasher, Goldstein and Toppino, 1977; Fazio et al., 2015). There Is a New Voter in the Room walks through those studies.
The machine has the same weak spot, by construction. A model answers a factual question better the more of its training documents mention it (Kandpal et al., 2023): frequency is a large part of how it learns what to say. In 2025 researchers found that about 250 poisoned documents planted a narrow backdoor, a trigger phrase that made the model output gibberish, in models from 600 million to 13 billion parameters, even though the largest trained on more than 20 times as much clean data (Souly et al., 2025, a preprint).
And I run the polite version for a living. The rule in my own GEO playbook is that one sentence, repeated in the same shape across eight or more places, beats fifty original posts. I Engineer Illusory Truth for Machines is the confession.
I have come to a simple conclusion. Only when an event happens and we observe it does probability harden into fact. But if your observation is fed by a biased source, your reality warps to match. Observation Is the Cut is the long version of that sentence.
Diagnosis: bias is a prior, not a bug
So when we complain that AI is biased, we miss the point. The AI is a mirror. You cannot easily code out bias, because bias is just context. It is the sum of every prior prompt that ever shaped the mind.
Machine learning has had a precise version of “intelligence requires bias” since 1980. Put three dots on a graph. Infinitely many curves pass through all three. To guess where the fourth dot goes, you have to prefer one curve, the smoothest, say, and nothing in the three dots tells you which. Tom Mitchell, the same Mitchell as the truth probe above, called that preference a bias: “any basis for choosing one generalization over another, other than strict consistency with the observed training instances”. A learner without one, he argued, can do no better on new cases than a lookup table of the old ones (Mitchell, 1980).
David Wolpert later proved a formal version for supervised learning: averaged over every possible problem, no learning method beats any other, so whatever edge a method has on a real problem comes from assumptions that fit it (Wolpert, 1996).
Mitchell meant bias in a technical sense: a preference among generalizations. Psychology had already built the bridge to prejudice. “The human mind must think with the aid of categories,” Gordon Allport wrote in 1954. “Once formed, categories are the basis for normal prejudgment. We cannot possibly avoid this process. Orderly living depends upon it” (Allport, 1954).
Ted’s prior is the same machinery that lets him cross a street without recomputing traffic from scratch. Fed a fair sample, we call it experience. Fed three blond Dimas, we call it prejudice.
Bias is not a flaw bolted onto intelligence. It is the price of having a context at all.
That does not make every bias fine, and it does not mean nothing can be done. It changes the question. The answer to “is it biased at all?” is always yes. The useful questions are which bias, learned from what, and whether we can still see it.
The dangerous fix: debiasing that only hides
The last part of that question matters most, because the obvious fix can make things worse.
In 2016 Tolga Bolukbasi and colleagues gave the problem its slogan, in their title: man is to computer programmer as woman is to homemaker. They found a gender direction in word vectors and removed it from words that should carry no gender, such as job titles (Bolukbasi et al., 2016). Measured along that direction, the bias was gone.
Three years later Hila Gonen and Yoav Goldberg looked again, and the words still kept their old company. Take the 500 most male-leaning and 500 most female-leaning words of the original space, sort the debiased versions into two clusters, and 92.5% still landed with their original gender, against 99.9% before. Their title says it: lipstick on a pig. Removing the projection, they wrote, “merely hides the bias” (Gonen and Goldberg, 2019). That was two debiasing methods on static word vectors.
Then the same thing in language models, with higher stakes. Valentin Hofmann and colleagues showed models the same content written in African American English and in Standardized American English, and asked about the writer. Asked directly about African Americans, the models answered with mostly positive words. Shown the dialect, they produced stereotypes more negative than any recorded in human studies of attitudes toward African Americans. In hypothetical decision tasks they matched the dialect’s writers with less prestigious jobs, convicted them more often (68.7% of the time, against 62.1%) and chose the death penalty more often (27.7% against 22.8%) (Hofmann et al., 2024).
Bigger models did not shed the covert prejudice, and training with human feedback made the overt answers nicer while the covert ones stayed. In the authors’ words, such training ends up “superficially obscuring the racism that language models maintain on a deeper level”. The trials had no real defendants, and only two of the models tested, GPT-3.5 and GPT-4, had been trained with human feedback.

Ted can learn this too. Shame him for saying “poison” out loud, and he will smile and take the cup. He still will not drink it.
In The Lock You Cannot Pick I wrote that models punished for what they show, in tests built to tempt them, hide better. This is the bias version of the same case. In both studies, the fix removed what the direct test could see, and the association stayed underneath, out of sight. My old notes have the human version: you can move, adopt a new culture, start taking off your shoes at home, and your original background still shapes how you reason. The fine-tune sits on top. The base stays.
The title needs its limit. Bias can be reduced, and measured. In an Anthropic study of its own models, a preprint, simply asking the largest model to avoid stereotypes cut its bias score on a standard benchmark by about 43%, and by about 84% when it was also asked to reason step by step. The same study found that, left unasked, the larger models were more biased than the small ones (Ganguli et al., 2023).
What I do not think anyone can do is remove bias altogether and keep a mind that generalizes. Mitchell’s argument covers the technical half. The social half is my reading: a language model learns its preferences from our text, and our text carries our associations. The debiasing to fear is the kind that makes bias invisible to the test you happened to run.
Treatment: a better diet
The fix is not censorship. It is diversity of data.
This is why I keep telling people to read across languages, to deliberately sample other perspectives. It frees your focus. My old notes put the reason in one line: it stops the event from becoming a fact.
Boaz Keysar, Sayuri Hayakawa and Sun Gyu An gave students the classic framing problem, where the same outcome of a disease outbreak is described as lives saved or lives lost. In their native language people flipped with the frame. In a foreign language they had learned, the flip disappeared (Keysar, Hayakawa and An, 2012). These were hypothetical vignettes, with a few dozen students per condition.
The record since then is mixed. Two meta-analyses find a small but reliable average foreign-language effect on decisions, mostly from moral dilemmas rather than framing (Circi et al., 2021; Del Maschio et al., 2022). But one of the largest framing tests, with over 1,500 people, found no effect of the foreign language itself (Oganian, Korn and Heekeren, 2016). And in two preregistered experiments, proficient bilinguals were worse at telling true news from false in their second language, where false headlines looked more believable (Muda et al., 2023).
Distance from your own language seems to loosen a frame’s grip on some decisions, and it can make you worse at spotting a false headline. What I think helps is the other perspective a second language opens: different sources, different authors, the story told from the other side.
For models there is a twist. When Anthropic researchers put the questions of two global opinion surveys to one of their own models, its default answers sat closer to opinions in the USA and some European and South American countries. Asking the same questions in Russian, Chinese or Turkish did not, by itself, move its answers toward the opinions of people who speak those languages (Durmus et al., 2024). Translation serves the same meal on a different plate.

So the prescription, case by case:
- Measure covert bias, not only overt. Ask about the group, then also show the model the group’s dialect, names and situations: same content, different guise, the way Hofmann’s team did. If only the direct question improves, you have taught the model what not to say.
- Feed it more than one perspective. Training and retrieval data from more than one culture, source and language, chosen on purpose. Translated copies of one source count once.
- Make it show its sources. Retrieval with citations turns a likelihood into a claim you can check against something outside the model.
- Treat a hidden bias as worse than a visible one. A bias you can measure, you can manage. One that only went quiet can surface in the case nobody tested.
- Check with a fresh mind. For models, a reviewer in a new context that reads the output against the source (The Infinite Review is how I keep mine from reviewing forever). For people, the same story from another source, in another language.
If we want this not-so-artificial intelligence to be better than us, and to trust the blond guy with the coffee, we should not lobotomize it. We should feed it a better diet.
Not so artificial.
Next: the question the skeptic keeps asking. Does it actually understand any of this, or is it just predicting the next word? The Vector opens the machinery.
Key takeaways
- Intelligence requires bias. A learner needs some basis for preferring one generalization over another beyond the data it has seen, or it cannot generalize at all. Ted’s prior and a model’s do the same job, fed different data.
- The mirror learned our associations. Word vectors trained on web text reproduced every implicit bias their authors tested from psychology, and across 50 occupations how strongly a job leaned female tracked the real share of women in it at a correlation of 0.90.
- Likely is not the same as true. Models carry a weak internal signal for the truth of simple statements, but the output is trained to follow likelihood: on a test of popular misconceptions, the largest 2021 models were generally the least truthful.
- Repetition re-weights the context. People rate repeated statements as truer, and knowing better does not reliably protect them; models learn what to say largely from frequency, and in a 2025 preprint about 250 poisoned documents were enough to plant a narrow backdoor.
- Debiasing can hide what it does not remove. Debiased word vectors still clustered by gender, and in the two models tested, training with human feedback left covert dialect prejudice under nicer overt answers.
- Another language may help through its sources. The foreign-language effect on decisions is small on average and uneven, bilinguals judged false news worse in their second language, and asking a model in another language did not by itself change whose opinions it echoed.
- Treatment: measure covert bias, feed the model more than one perspective (translated copies of one source count once), make it show its sources, treat a hidden bias as worse than a visible one, and review with a fresh mind.
Sources
- Allport, G. W. (1954). The Nature of Prejudice, ch. 2, p. 20. Cambridge, MA: Addison-Wesley. https://archive.org/details/in.ernet.dli.2015.188638
- Azaria, A., & Mitchell, T. (2023). “The Internal State of an LLM Knows When It’s Lying.” Findings of EMNLP 2023: 967-976. (Measures whether a probe can read statement truth; two models, OPT-6.7b and LLaMA2-7b.) https://doi.org/10.18653/v1/2023.findings-emnlp.68
- Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. T. (2016). “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.” Advances in Neural Information Processing Systems 29. https://arxiv.org/abs/1607.06520
- Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2023). “Discovering Latent Knowledge in Language Models Without Supervision.” ICLR 2023. (The misleading-prompt test moved one model, UnifiedQA.) https://arxiv.org/abs/2212.03827
- Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). “Semantics derived automatically from language corpora contain human-like biases.” Science 356(6334): 183-186. https://doi.org/10.1126/science.aal4230
- Circi, R., Gatti, D., Russo, V., & Vecchi, T. (2021). “The foreign language effect on decision-making: A meta-analysis.” Psychonomic Bulletin & Review 28(4): 1131-1141. https://doi.org/10.3758/s13423-020-01871-z
- Del Maschio, N., Crespi, F., Peressotti, F., Abutalebi, J., & Sulpizio, S. (2022). “Decision-making depends on language: A meta-analysis of the Foreign Language Effect.” Bilingualism: Language and Cognition 25(4): 617-630. https://doi.org/10.1017/S1366728921001012
- Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., et al. (2024). “Towards Measuring the Representation of Subjective Global Opinions in Language Models.” COLM 2024 (company paper, Anthropic; one model). https://arxiv.org/abs/2306.16388
- Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015). “Knowledge does not protect against illusory truth.” Journal of Experimental Psychology: General 144(5): 993-1002. https://doi.org/10.1037/xge0000098
- Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., et al. (2023). “The Capacity for Moral Self-Correction in Large Language Models.” arXiv 2302.07459 (preprint; company paper, Anthropic). https://arxiv.org/abs/2302.07459
- Gonen, H., & Goldberg, Y. (2019). “Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them.” Proceedings of NAACL-HLT 2019: 609-614. https://doi.org/10.18653/v1/N19-1061
- Hasher, L., Goldstein, D., & Toppino, T. (1977). “Frequency and the conference of referential validity.” Journal of Verbal Learning and Verbal Behavior 16(1): 107-112. https://doi.org/10.1016/S0022-5371(77)80012-1
- Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024). “AI generates covertly racist decisions about people based on their dialect.” Nature 633(8028): 147-154. (Hypothetical tasks.) https://doi.org/10.1038/s41586-024-07856-5
- Kandpal, N., Deng, H., Roberts, A., Wallace, E., & Raffel, C. (2023). “Large Language Models Struggle to Learn Long-Tail Knowledge.” Proceedings of ICML 2023, PMLR 202: 15696-15707. https://arxiv.org/abs/2211.08411
- Keysar, B., Hayakawa, S. L., & An, S. G. (2012). “The foreign-language effect: Thinking in a foreign tongue reduces decision biases.” Psychological Science 23(6): 661-668. https://doi.org/10.1177/0956797611432178
- Levinstein, B. A., & Herrmann, D. A. (2024). “Still no lie detector for language models: probing empirical and conceptual roadblocks.” Philosophical Studies 182(7): 1539-1565 (online 2024, issue 2025). https://doi.org/10.1007/s11098-023-02094-3
- Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023). “Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.” Advances in Neural Information Processing Systems 36. https://arxiv.org/abs/2306.03341
- Lin, S., Hilton, J., & Evans, O. (2022). “TruthfulQA: Measuring How Models Mimic Human Falsehoods.” Proceedings of ACL 2022: 3214-3252. https://doi.org/10.18653/v1/2022.acl-long.229
- Marks, S., & Tegmark, M. (2024). “The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets.” COLM 2024. https://arxiv.org/abs/2310.06824
- Mitchell, T. M. (1980). “The Need for Biases in Learning Generalizations.” Technical Report CBM-TR-117, Rutgers University. https://www.cs.cmu.edu/~tom/pubs/NeedForBias_1980.pdf
- Muda, R., Pennycook, G., Hamerski, D., & Białek, M. (2023). “People are worse at detecting fake news in their foreign language.” Journal of Experimental Psychology: Applied 29(4): 712-724. https://doi.org/10.1037/xap0000475
- Oganian, Y., Korn, C. W., & Heekeren, H. R. (2016). “Language switching, but not foreign language use per se, reduces the framing effect.” Journal of Experimental Psychology: Learning, Memory, and Cognition 42(1): 140-148. https://doi.org/10.1037/xlm0000161
- Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu, B., et al. (2025). “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples.” arXiv 2510.07192 (preprint). https://arxiv.org/abs/2510.07192
- Wolpert, D. H. (1996). “The Lack of A Priori Distinctions Between Learning Algorithms.” Neural Computation 8(7): 1341-1390. https://doi.org/10.1162/neco.1996.8.7.1341
Related reading: The Loop of Attention is the step before this one: why a mind that loses its thread fills the gap with a confident guess. The Vector opens up the machinery underneath: if a fact and a belief are both high-probability links between vectors, it helps to know what a vector is. The Infinite Review traces one bias end to end, from human idleness aversion to the machine reviewer that always finds something. The Lock You Cannot Pick is the same hiding in safety training, seen in tests built to tempt the models. There Is a New Voter in the Room follows repetition from one mind into a whole society.
Common Questions
- Can AI ever be unbiased?
- Not in the sense of having no bias at all. Any system that learns has to prefer some generalizations over others beyond the data it has seen, which machine learning calls inductive bias, and a model trained on human text also inherits human associations. What can be done is to choose and measure the biases: harmful ones can be reduced, and the useful test is whether the bias is visible or merely hidden.
- Why is AI biased?
- Because it learns from human text, and the statistics of human language carry human associations. A 2017 study in Science found that word vectors trained on a large web crawl reproduced every implicit bias the authors tested from psychology, including links between gendered names and career or family. Whose views dominate may matter too: one study found a model's default answers on global opinion surveys closer to some countries' views than others'.
- Does debiasing AI work?
- Partly. Bias can be reduced and measured, but some methods hide it rather than remove it. After a popular debiasing method, word vectors still clustered by gender, and a 2024 Nature study found that the two language models in it trained with human feedback, GPT-3.5 and GPT-4, gave more positive overt answers about African Americans while covert prejudice triggered by African American English remained. Test for covert bias, not only for what the model says when asked directly.
- Does an AI know what is true?
- In a weak sense, sometimes. Interpretability studies find directions inside language models that track whether simple statements are true, and steering a model along them made its answers more truthful in tests. But the output is trained to follow likelihood and does not have to obey that signal, and early truth probes failed to generalize in basic ways, so there is no reliable lie detector inside a model yet.
- Why does propaganda work on people and on AI?
- Because both learn from repetition. People rate statements they have seen before as more likely true, and knowing better does not reliably protect them, which psychologists call the illusory truth effect. Language models learn what to say largely from how often things appear in their training data, and a 2025 preprint found that about 250 poisoned documents were enough to plant a narrow backdoor, a trigger that made the model output gibberish, in models up to 13 billion parameters that the researchers trained themselves.
- Does thinking in a foreign language reduce bias?
- Sometimes, for some decisions, and not reliably. Two meta-analyses find a small but reliable average effect, mostly from moral dilemmas, but one of the largest framing tests found no effect of the foreign language itself, and bilinguals were worse at telling true news from false in their second language. In my view, the useful part of reading in another language is the other sources it opens up.
- What is inductive bias in machine learning?
- Inductive bias is whatever makes a learner prefer one generalization over another when several fit the data equally well, such as a preference for simpler or smoother explanations. Tom Mitchell argued in 1980 that a learner without any such bias can do no better on new cases than a lookup table of the old ones, and the no-free-lunch theorems later showed that no learning method beats another when averaged over all possible problems. Bias in this sense is what makes generalizing possible, and the question is whether it fits the problem.