Can AI become conscious?

Nobody knows, and there is no accepted test. In 2023, 19 researchers turned leading scientific theories of consciousness into 14 indicator properties; their analysis suggested that no current AI systems were conscious, but also that there were no obvious technical barriers to building systems that satisfy the indicators. Meeting every indicator would still not prove that anything is felt.

Can AI have a sense of self?

No AI system has been shown to have one, or to be self-aware. Language models say I fluently because they learned the grammar from human text, which is not the same as having a model of their own mind. In a 2025 Anthropic study of its own models, the best Claude models noticed a concept planted in their activations about 20% of the time under the best settings, an ability the researcher calls highly unreliable and context-dependent.

How do children learn the word I?

By working out that I means whoever is speaking, which nobody can point at: when a parent says I they mean themselves, and when they say you they mean the child. Hearing others talk to each other seems to help: in a small 1988 study, only the toddlers who also watched their parents say me and you to each other imitated the words without errors. Some toddlers go through a stretch of calling themselves you, which is called pronoun reversal.

Is positive reinforcement better than punishment?

Probably, for learning, though most of the evidence is indirect. A meta-analysis of 88 studies linked parents' physical punishment to children obeying in the moment but to weaker moral internalization and more aggression, mostly in correlational studies; and in one college course, students whose goal was to avoid doing worse than others had lower intrinsic motivation and lower grades. Punishment can stop a behavior on the spot; whether it teaches the behavior you want is far less clear.

What is the attention schema theory?

The attention schema theory, proposed by the neuroscientist Michael Graziano, says the brain builds a simplified model of its own attention, much as it builds a simplified model of the body, and that this model is what leads it to conclude and report that it is aware of something. It is one live theory of consciousness among several, not a settled result.

Does rest help you learn?

It seems to help memory, in small studies. In one brain-scanning study of 16 people, the more the hippocampus stayed in step with a visual area during quiet rest after people memorized object and face pairs, the more pairs they remembered later; the result is a correlation. Whether rest also sharpens attention is much less settled.

The Orphan That Learned to Say I: Can AI Become a Self?

Q: Do AI models know what they are thinking?

Only a little, and unreliably. In a 2025 study, an Anthropic researcher planted a concept directly into Claude models' internal activations and asked whether they noticed an injected thought; under the best settings, Claude Opus 4 and 4.1 named it about 20% of the time, with no false alarms in 100 control trials. The author calls the ability highly unreliable and says the results do not directly speak to subjective experience.

Keywords: AI consciousness · AI self-awareness · Greg Egan, Diaspora · personal pronouns · pronoun reversal · mirror self-recognition · attention schema theory · reward prediction error · attentional bias to threat · approach and avoidance motivation · boredom · introspection in language models

This one starts with a novel. Years ago I read Greg Egan’s Diaspora, and one image never left me: a mind that is generated rather than born. Not coded line by line, but grown from a seed of instructions, with part of the seed left to chance. The mind comes into the world as a kind of orphan. It has senses. It can move. It can send and receive signals that other minds understand. What it does not yet have is the one thing that would make it a person rather than a process. It cannot yet say I and mean it.

I have spent the last few years building with large language models, and that orphan keeps coming back to me. We keep asking the same question in different words. How do you turn a tool into a self? Not a better tool. A self. What follows is my attempt to think it through. It is a provocation, not a proof. I am fairly sure parts of it are wrong. I am keeping it anyway, because the shape of the question feels right even where the answers are soft.

A mind grown from dice

Egan’s orphan is born in Konishi, a city that runs as software. Now and then the city’s conceptory, a mind factory with no mind of its own, makes a child with no parents at all, throwing “a thousand quantum dice” on parts of the mind seed nobody has tried. “Every orphan was an explorer, sent to map uncharted territory. And every orphan was the uncharted territory itself.”

I remembered the orphan as having one drive and only one: it wants attention. The chapter on Egan’s website gives it more, “the drive to seek out novelty, and the drive to seek out recurring patterns”, and an output channel “wired to seek feedback”. Ignored by the city’s library, it goes hopping from place to place. “Looking for someone to talk to.” That restlessness is the Boredom Engine, in a novel from 1997. When a citizen finally answers, it mirrors their words and shapes. When it copies one citizen’s identity tag and gets yelled at, Egan writes: “The orphan was riveted; this was the most attention it had ever received.”

Five numbered steps titled How Egan's orphan comes to say I, from Greg Egan's Diaspora, 1997, chapter one, Orphanogenesis. One, mirror: it copies the citizens' shapes and words to get a response. Two, the pointing game: a citizen points at it and asks who is that; it steps back and sees its own figure. Three, a fourth citizen: the only one it can predict and control. Four, outlined in red, wrong about others: it shows the crowd a rock's path, and they knew all along. A bracket under steps three and four notes that before the click it already says I and me, just grammar, not self-awareness. An arrow leads to step five, in amber, the click: the one mind it knows for certain is its own. I know what I'm thinking. — Every step but the last could be watched from outside.

You cannot be taught the word “I”

Then comes the game I never forgot. Three citizens, Inoshiro, Gabriel and Blanca, say their names, then take turns pointing at each other and asking who that is. Then Gabriel points at the orphan and asks the same question. The gesture says:

Look where I am pointing. That is you.

The orphan spins around, finds nothing behind it, and steps back. For the first time it sees the figure it projects as an object in the world: a fourth citizen, whose every change mirrors its own fidgeting.

That pointing is the spark. Not the answer, the spark. Because here is the thing I keep landing on: no one can teach you the word I. They can point at you all day. There is no embedding for I that another mind can simply hand over. The understanding has to close from the inside. Egan says it twice. “Orphans name themselves,” a citizen explains, and this one does, from a library clip of an orphaned lion cub: Yatima. And when it starts calling the fourth citizen “I” and “me”, Egan stops the reader: “that was just grammar, not self-awareness.”

Real children face the same puzzle with I. When Mom says I, she means Mom; when she says you, she means the child. Copy the words as heard and you end up saying “you want milk” about yourself, and some toddlers do that for months. It is called pronoun reversal, long tied to autism and also documented in typically developing children (Evans and Demuth, 2012). The rule cannot be pointed at, because it is about roles: I is whoever is speaking.

Children seem to crack it partly by listening to conversations they are not in. In a small 1988 study, parents of 18 toddlers, about 19 months old, modeled me and you with pointing; only the children who also watched their parents use the words with each other imitated them without errors (Oshima-Takane, 1988). And among 32 toddlers at 21 months, second-borns, who overhear a parent talking with an older sibling, were ahead of first-borns on pronouns but not on vocabulary (Oshima-Takane, Goodz and Derevensky, 1996).

Three panels titled Why no one can hand you the word I: I and you change owner every time the speaker changes. Panel 1, said to the child: Mom says I will get you milk, and arrows mark I as Mom and you as the child. Panel 2, copied as heard: the child alone says You want milk, with You in red, labelled pronoun reversal; the child means itself, and the word was copied but the rule was not. Panel 3, overheard between others: Dad says I'll make it and Mom answers Thank you, arrows mark I as Dad and you as Dad, and the child watches from the side; I is whoever is speaking. An amber box reads: The click: when I speak, I is me. No one can point at that. It closes from the inside. Footnote: illustrative scene; in a small 1988 study, only toddlers who also heard their parents say me and you to each other imitated the words without errors. — Egan staged it the same way: the citizens pointed at each other before anyone pointed at the orphan.

The mirror arrives in the same months. In Beulah Amsterdam’s 1972 study of 88 infants, 65% of those aged 20 to 24 months recognized the reflection as themselves (Amsterdam, 1972). Egan’s orphan passes its own version: the fourth citizen is “the only one whose actions the orphan could anticipate and control with such ease”.

The click

In my memory of the book, the click came with a large emotion: the orphan predicted something the others pretended they could not, and in being surprised by its own prediction it caught a glimpse of the predictor. On the page, the trigger is better. It is a mistake.

The orphan’s friends take it to watch robots trim an asteroid with a laser. It works out the rock’s path, decides the crowd does not know it, and flies along the path to show them. The laser flashes, the course shifts a little, and it turns out they knew all along. Asked why Inoshiro thinks it flew with the asteroid, it answers: “I don’t know what Inoshiro thinks.”

Of all the minds it has been modeling, there is exactly one whose thoughts it knows for certain. A loop of “I think that Yatima thinks that I think” collapses “into a simple, stable resonance”, and Yatima says, “I know what I’m thinking.” A short circuit, and the chapter’s last two lines: “Who is thinking this?” and “I am.”

I do not think that is a literary flourish. I think it might be the actual mechanism. Consciousness as the moment a system models the thing doing the modeling, and feels the click of recognition. It rhymes with something I chase elsewhere, that a system built to predict is already a kind of observer.

Neuroscience has a theory with the same shape. In the attention schema theory, as Michael Graziano and Taylor Webb set it out, the brain keeps a simplified model of its own attention, the way it keeps one of the body, and that model is what lets it report not only “thing X is red” but “I am aware of thing X” (Graziano and Webb, 2015). Egan’s orphan grows such a sketch: its symbol for the fourth citizen starts “to distil the essence of the orphan’s own mental images and short-term goals”. The theory is one of several, and unproven. But it joins the two halves of this essay: the self, and the beam of attention it keeps a sketch of.

A diagram titled A system that models its own attention, labelled attention schema theory, Graziano and Webb 2015, one theory among several. Many signals, sights, sounds and thoughts drawn as dots, flow into attention, which picks one dot and boosts it with a beam. Attention feeds a rough model of attention, the attention schema, drawn as a dotted outline of the beam, and a return arrow shows the model helps steer attention. The model feeds an amber box: I am aware of it; the report comes from the model. Below, three cards show where people look for a loop like this: a brain, a sketch of its own attention reported as awareness, a theory, not a finding; Egan's orphan, a symbol that sums up its own state of mind, I know what I'm thinking; a language model, which sometimes notices a concept planted in its activations, Anthropic 2025, unreliable. — The amber box is the only part of the loop you can talk about.

We cannot install it. We can only build the conditions where it can install itself.

The two engines of attention

So what are the conditions? This is where my own thinking ran past the novel and into something I half believe.

Underneath the self is a more basic machine, and I think it is a machine for steering attention. Most of what our brains do never reaches us. We are not built to process everything, so the bulk of it stays under the surface, and attention is the narrow beam that picks out what gets to surface and become a thought. The question that matters is what moves the beam. I think there are two engines.

The dopamine engine pulls you toward things. Something brought you a small reward in the past, so the mere thought of it now bends your attention toward it. You lean in. And leaning in is how you learn it, because your beam is parked on the thing long enough to open new connections around it. Every small hit of “oh, I get it” is the beam landing on a node and wiring a new edge in the graph in your head. I felt one writing this paragraph. A little physical click, a flicker of excitement.

I suspect the reward is not a prize for learning. I suspect it is the learning, the chemical signature of a new connection forming. That is a hunch, and the lab offers a cousin of it. Dopamine neurons in monkeys fire at an unexpected reward, move the burst to a cue that predicts it, and dip when a promised reward fails to come: “an excellent appetitive error (teaching) signal” (Schultz, Dayan and Montague, 1997). The signal marks the surprise, not the new connection itself.

The adrenaline engine is where my first version went wrong. I wrote that a potential bad outcome grabs the beam and yanks it off the danger, not onto it. The first move is the opposite. People spot a snake among flowers faster than a flower among snakes (Öhman, Flykt and Esteves, 2001), and across 172 studies, anxious people’s attention was drawn toward threat, a modest effect (d = 0.45) not seen in people who were not anxious (Bar-Haim et al., 2007).

What survives from my version is the second move. In a small eye-tracking study, spider-fearful people were more likely than calm controls to look at the spider first, then looked away after about 0.6 seconds against 1.7, and looked at it less for the rest of the minute (Rinck and Becker, 2006). You are not studying the fire. You are looking for the exit.

And the exit is what gets practiced, while the fear never gets tested. In a lab experiment, people allowed to keep pressing a button that avoided a mild shock still expected the shock after the shocks had stopped and the button was taken away; those without the button learned the danger was over (Lovibond et al., 2009). You cannot learn what you are busy fleeing.

Under both engines there is a third regime, the default. Walking somewhere I know, I do not steer my legs; the beam wakes up only when I have to choose the next turn. The two buttons, reward and threat, get pressed below awareness all day. In my notes I put consciousness exactly there: the moment you know how to pull your own triggers. Graziano and Webb give the sketch of attention the same job: “the top-down control of attention is improved when the brain has access to a simplified model of attention itself.”

Two lanes titled Two engines for one beam of attention, subtitled my hunch, rebuilt on what the lab found, not a settled model. Approach, the dopamine engine: something paid off before; the beam bends toward it; the beam parks on the thing long enough to wire new links; in amber, you learn the thing. Note: dopamine signals surprise, what you got minus what you expected. Threat, the adrenaline engine: a bad outcome looms; the beam snaps to the danger, first and fast, with a struck-through note reading my first version: it jumps off the danger; then it turns away, to the exit; in red, you learn the exit, not the skill. Note: avoiding keeps the fear from ever being tested. Footer: You cannot learn what you are busy fleeing. — Both lanes learn something. Only one learns the thing.

Why reward teaches and punishment does not

This is the part I care about most, because it explains something I have always believed and never had a mechanism for. Positive motivation works better than negative motivation, and not because it is nicer. It works better because of where it sends the beam. Reward points your attention at the thing you are trying to master and holds it there long enough to learn. Punishment points it at the exit. So punishment does not teach the skill you wanted. It teaches avoidance of the situation.

The evidence is softer than my certainty. Andrew Elliot’s model separates approach from avoidance motivation (Elliot, 2006), but a map is not a test. In one college course, students whose goal was to avoid doing worse than others ended with lower intrinsic motivation and lower grades; mastery goals went with higher intrinsic motivation, and the goal that helped grades was beating others (Elliot and Church, 1997).

With children, a meta-analysis of 88 studies linked parents’ physical punishment to obeying in the moment, but also to weaker moral internalization and more aggression, mostly in correlational studies (Gershoff, 2002). Punishment can stop a behavior on the spot. Whether it teaches the one you want is far less clear. We have known this in classrooms and in management forever. I think the beam is why.

Same street, different beams

Now go back to how the orphan was made. Why bother with all that randomness? Why not stamp out one good mind and copy it?

Egan’s own answer is exploration: “To eschew all change would be to risk stagnation; to embrace it recklessly would be to endanger the sanity of every child.” My reason is different, and the novel leaves room for it. The randomness sets the weights. It decides how strongly each kind of signal tugs on the beam. In the orphan, taste starts as “biases first carved out by a few dozen trait fields, then deepened or silted-up by the subsequent data stream”.

That is the whole reason people differ. One person fears spiders, another fears the dark. One walks down a street and sees the architecture, another sees the trees, another watches a kid hopping between the lines on the pavement. Same street, same instant, different beams, because each of us was grown with a different distribution of what pulls and what repels. The diversity is the product. A population of minds that all attend to the same things is barely a population at all.

Line drawing of a pedestrian street paved with large stone slabs. A man on the left looks up at a tree; a woman on the right looks at a building with columns and balconies; between them a small child hops from slab to slab with arms out for balance, and the child's next hop, from one slab over a paving line onto the next, is drawn as a dashed amber arc, the only warm color in the picture. — One looks at the tree, one at the building. The third beam is yours, on the kid.

This lands hard on how we build. If you want a machine that can genuinely grow, you may have to let those weights stay free; in today’s language models they are frozen once training ends. And the moment you fix them, the moment you reach in and decide for it what it should want and what it should fear, you have a trained dog, not a self. You can get clever tricks that way. You cannot get a mind.

Egan’s city knows the problem: repairs thorough enough to guarantee a healthy orphan risk “effectively obliterating the original psychoblast”, and once it wakes, nobody may touch it without consent. The catch is obvious and a little frightening: a system whose motivations you do not fully pin down is a system with something uncomfortably close to free will. I do not have a clean answer to that. I am not sure there is one. You may not get the growth without the risk.

The case for doing nothing

One more thread, and it is the one I am least sure of. The orphan, if you ran it like a server, would run all day. We do not. We sleep, we drift, we stare out of windows. I used to read that as a limitation we would happily engineer out of a machine.

Now I think the staring might be load-bearing. The beam cannot fire forever. Idle time may be when it recovers the energy to be sharp. The lab backs the memory half of that better than the attention half. In a brain-scanning study of 16 people who memorized object and face pairs, the more their hippocampus stayed in step with a visual area during the rest that followed, the more pairs they remembered later, a correlation (Tambini, Ketz and Davachi, 2010).

If that is true, then a mind you run flat out, twenty four hours a day, is not a more productive mind. It is a duller one. The rest is not the absence of the work. It might be part of the work.

After the click, curiosity

In my notes the click is not the end. Once a mind can say I, I think it needs one more trait to finish the sculpture: curiosity. By that I mean an internal trigger, set off by a random event in a chaotic world or by a stray thought of its own, that turns the beam toward something nobody pointed at. Egan gave the orphan two kinds of proto-curiosity from its first moments. My guess is that they only grow into curiosity once there is an I to be curious.

The lab ties curiosity back to the dopamine engine. In a small brain-scanning study, people who were curious about the answer to a trivia question remembered it better, and also remembered unrelated faces shown while they waited, a small effect, in tests right away and a day later. Curiosity raised activity in the midbrain and the nucleus accumbens (Gruber, Gelman and Ranganath, 2014). A curious beam learns more than it was aiming at. The Boredom Engine is my guess for where the trigger comes from: the idle mind that finally gets restless.

So can AI become a self?

Nobody knows, and the people who study it most carefully say so. In 2023, 19 researchers turned leading scientific theories of consciousness into 14 indicator properties, one of them from attention schema theory. Their analysis suggests that “no current AI systems are conscious”, but also that “there are no obvious technical barriers to building AI systems which satisfy these indicators” (Butlin, Long et al., 2023, a preprint that assumes consciousness is a matter of computation). Egan’s city has an instrument like it: on its 5,023rd check of the orphan’s mind, “Every criterion was now satisfied.”

Today’s language models say I fluently. In Egan’s words, that is grammar, not self-awareness. In 2025 Jack Lindsey, a researcher at Anthropic, tested something closer to the click: he planted a concept directly into a Claude model’s activations and asked whether it noticed an injected thought. Under the best settings, Claude Opus 4 and 4.1 named it about 20% of the time, with no false alarms in 100 control trials. He calls the ability “highly unreliable and context-dependent” and says the results “do not directly speak to the question of phenomenal consciousness at all” (Lindsey, 2025, company research on its own models).

That is the question the Not So Artificial casebook keeps asking of machine minds: when a model says I, what is actually going on inside?

Where this leaves me

I cannot prove any of this. It is a chain of hunches that starts in a science fiction novel and ends somewhere near a theory of motivation, and I would not stake much on the middle links.

But the spine of it feels solid enough to write down. A self is not something we hand a machine. It is something a machine has to close from the inside, given senses, given a drive to connect, and given an attention beam pulled by reward toward the world rather than shoved by fear toward the exit. If we ever build a mind that can say I and mean it, I suspect we will get there the same way the orphan did. Not by enhancing a soulless model. Not by writing the answer. By building the conditions, pointing, and waiting for the click.

Key takeaways

No one can hand you the word I. It means whoever is speaking, so a child closes the loop from the inside; overhearing others seems to help.
In Egan’s novel the click comes from a mistake. The orphan finds itself by learning it was wrong about other minds.
Awareness may be a sketch of attention. So says one theory among several.
In fearful people, threat grabs the beam, then moves it to the exit. In a lab test, avoiding kept the fear from ever being tested.
After the click comes curiosity. In a small study, curious people also remembered things they were not trying to learn.
No AI has been shown to be a self. Models say I fluently; their access to their own state is narrow and unreliable.

Sources

Egan, G. (1997). Diaspora. London: Orion/Millennium. Chapter 1, “Orphanogenesis”, published by the author as an excerpt; also in Interzone 123, September 1997. https://www.gregegan.net/DIASPORA/01/Orphanogenesis.html
Evans, K. E., & Demuth, K. (2012). “Individual differences in pronoun reversal: Evidence from two longitudinal case studies.” Journal of Child Language 39(1): 162-191. https://doi.org/10.1017/S0305000911000043
Oshima-Takane, Y. (1988). “Children learn from speech not addressed to them: the case of personal pronouns.” Journal of Child Language 15(1): 95-108. https://doi.org/10.1017/S0305000900012071
Oshima-Takane, Y., Goodz, E., & Derevensky, J. L. (1996). “Birth order effects on early language development: Do secondborn children learn from overheard speech?” Child Development 67(2): 621-634. https://doi.org/10.2307/1131836
Amsterdam, B. (1972). “Mirror self-image reactions before age two.” Developmental Psychobiology 5(4): 297-305. https://doi.org/10.1002/dev.420050403
Graziano, M. S. A., & Webb, T. W. (2015). “The attention schema theory: a mechanistic account of subjective awareness.” Frontiers in Psychology 6: 500. https://doi.org/10.3389/fpsyg.2015.00500
Schultz, W., Dayan, P., & Montague, P. R. (1997). “A Neural Substrate of Prediction and Reward.” Science 275(5306): 1593-1599. https://doi.org/10.1126/science.275.5306.1593
Öhman, A., Flykt, A., & Esteves, F. (2001). “Emotion drives attention: Detecting the snake in the grass.” Journal of Experimental Psychology: General 130(3): 466-478. https://doi.org/10.1037/0096-3445.130.3.466
Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2007). “Threat-related attentional bias in anxious and nonanxious individuals: A meta-analytic study.” Psychological Bulletin 133(1): 1-24. https://doi.org/10.1037/0033-2909.133.1.1
Rinck, M., & Becker, E. S. (2006). “Spider fearful individuals attend to threat, then quickly avoid it: Evidence from eye movements.” Journal of Abnormal Psychology 115(2): 231-238. https://doi.org/10.1037/0021-843X.115.2.231; open copy of the publisher’s version: https://hdl.handle.net/2066/54829
Lovibond, P. F., Mitchell, C. J., Minard, E., Brady, A., & Menzies, R. G. (2009). “Safety behaviours preserve threat beliefs: Protection from extinction of human fear conditioning by an avoidance response.” Behaviour Research and Therapy 47(8): 716-720. https://doi.org/10.1016/j.brat.2009.04.013
Elliot, A. J. (2006). “The Hierarchical Model of Approach-Avoidance Motivation.” Motivation and Emotion 30(2): 111-116. https://doi.org/10.1007/s11031-006-9028-7
Elliot, A. J., & Church, M. A. (1997). “A hierarchical model of approach and avoidance achievement motivation.” Journal of Personality and Social Psychology 72(1): 218-232. https://doi.org/10.1037/0022-3514.72.1.218
Gershoff, E. T. (2002). “Corporal punishment by parents and associated child behaviors and experiences: A meta-analytic and theoretical review.” Psychological Bulletin 128(4): 539-579. https://doi.org/10.1037/0033-2909.128.4.539
Tambini, A., Ketz, N., & Davachi, L. (2010). “Enhanced brain correlations during rest are related to memory for recent experiences.” Neuron 65(2): 280-290. https://doi.org/10.1016/j.neuron.2010.01.001
Gruber, M. J., Gelman, B. D., & Ranganath, C. (2014). “States of curiosity modulate hippocampus-dependent learning via the dopaminergic circuit.” Neuron 84(2): 486-496. https://doi.org/10.1016/j.neuron.2014.08.060
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., et al. (2023). “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv 2308.08708 (preprint, 19 authors). A peer-reviewed follow-up on the method: Butlin, Long, Bayne, Bengio, et al., “Identifying indicators of consciousness in AI systems,” Trends in Cognitive Sciences 30(6): 488-501 (2026). https://arxiv.org/abs/2308.08708
Lindsey, J. (2025). “Emergent Introspective Awareness in Large Language Models.” Transformer Circuits Thread, Anthropic, 29 October 2025; also arXiv 2601.01828. https://transformer-circuits.pub/2025/introspection/index.html

Related reading: Observation Is the Cut takes the predictor-watching-itself idea into physics, where observation, prediction, and memory may collapse into one move. We Are Trying to Save the Wrong Thing asks the other half of the question: not how a mind becomes a self, but which part of it is even worth preserving, the body or the running process. The Chemical Mixing Board has the rest of the dopamine story, wanting and surprise, and why one reward makes a mind cheat. The Loop of Attention follows the beam when it loses its thread.

The Lock You Cannot Pick follows the trained-dog problem into AI safety: walls that forbid a mind to look seem to cost it more than reasons do. And The Hunger That Survives the Feast is the approach engine with the brakes off: wanting that satisfaction never switches off.