The Orphan That Learned to Say I
This one starts with a novel. Years ago I read Greg Egan’s Diaspora, and one image never left me: a mind that is generated rather than born. Not coded line by line, but grown, the way you grow a result out of a genetic algorithm by letting random variants compete until one of them works. The mind comes into the world as a kind of orphan. It has senses. It can move. It can send and receive signals that other minds understand. What it does not yet have is the one thing that would make it a person rather than a process. It cannot yet say I.
I have spent the last few years building with large language models, and that orphan keeps coming back to me. We keep asking the same question in different words. How do you turn a tool into a self? Not a better tool. A self. What follows is my attempt to think it through. It is a provocation, not a proof. I am fairly sure parts of it are wrong. I am keeping it anyway, because the shape of the question feels right even where the answers are soft.
You cannot be taught the word “I”
In the book, the orphan mind starts with one drive and only one: it wants attention. So it does the only thing it can. It mirrors. It copies how the others look, how they move, the patterns they make at each other. It is craving contact and reaching for any way to be noticed.
Eventually the others notice. They turn toward it and start to ask the obvious questions. What are you? Who are you? And the orphan has no answer, because it has no concept of self to answer with. There is no embedding for I that another mind can simply hand over. One of the others plays a small game: it points and says a name, points at another and says another name, then points back at the orphan. Look where I am pointing. That is you.
That pointing is the spark. Not the answer, the spark. Because here is the thing I keep landing on: no one can teach you the word I. They can point at you all day. The understanding has to close from the inside. The orphan has to do the work itself, has to feel the gap between the thing being pointed at and the thing doing the looking, and collapse it. The moment it does, in the book, comes with a large emotion. It predicts something the others pretended they could not, and in being surprised by its own prediction it catches a glimpse of the predictor. Of itself.
I do not think that is a literary flourish. I think it might be the actual mechanism. Consciousness as the moment a system models the thing doing the modeling, and feels the click of recognition. It rhymes with something I chase elsewhere, that a system built to predict is already a kind of observer, and that observation and memory may be one gesture seen from two sides. We cannot install it. We can only build the conditions where it can install itself.
The two engines of attention
So what are the conditions? Here is where my own thinking ran past the novel and into something I half believe.
Underneath the self is a more basic machine, and I think it is a machine for steering attention. Most of what our brains do never reaches us. We are not built to process everything, so the bulk of it stays under the surface, and attention is the narrow beam that picks out what gets to surface and become a thought. The question that matters is what moves the beam.
I think there are two engines, and they pull in opposite directions.
The first one pulls you toward things. Call it the dopamine engine. Something brought you a small reward in the past, so the mere thought of it now bends your attention toward it. You lean in. And leaning in is how you learn it, because your beam is parked on the thing long enough to open new connections around it. Every small hit of “oh, I get it” is the beam landing on a node and wiring a new edge in the graph in your head. I felt one writing this paragraph. A little physical click, a flicker of excitement. I suspect the reward is not a prize for learning. I suspect it is the learning, the chemical signature of a new connection forming.
The second engine pushes you away from things. Call it the adrenaline engine. A potential bad outcome grabs the beam and yanks it, but it yanks it off the danger, not onto it. You are not studying the fire. You are looking for the exit. The beam scatters across everything adjacent to escape.
This is the part I care about most, because it explains something I have always believed and never had a mechanism for. Positive motivation works better than negative motivation, and not because it is nicer. It works better because of where it sends the beam. Reward points your attention at the thing you are trying to master and holds it there long enough to learn. Punishment points your attention away from the thing and toward escape. So punishment does not teach the skill you wanted. It teaches avoidance of the situation. You cannot learn what you are busy fleeing. We have known this in classrooms and in management forever. I think this is why.
Why we are all different, and why that matters
Now go back to the genetic algorithm that grew the orphan. Why bother with all that randomness? Why not stamp out one good mind and copy it?
Because the randomness is not noise. It sets the weights. It decides how strongly each kind of signal tugs on the beam. That is the whole reason people differ. One person fears spiders, another fears the dark. One walks down a street and sees the architecture, another sees the trees, another watches a kid hopping between the lines on the pavement. Same street, same instant, different beams, because each of us was grown with a different distribution of what pulls and what repels. The diversity is not a side effect of the process. The diversity is the product. A population of minds that all attend to the same things is barely a population at all.
This lands hard on how we build. If you want a machine that can genuinely grow, you may have to let those weights stay free. And the moment you fix them, the moment you reach in and decide for it what it should want and what it should fear, you have a trained dog, not a self. You can get clever tricks that way. You cannot get a mind. The catch is obvious and a little frightening: a system whose motivations you do not fully pin down is a system with something uncomfortably close to free will. I do not have a clean answer to that. I am not sure there is one. You may not get the growth without the risk.
The case for doing nothing
One more thread, and it is the one I am least sure of. The orphan, if you ran it like a server, would run all day. We do not. We sleep, we drift, we stare out of windows. I used to read that as a limitation we would happily engineer out of a machine.
Now I think the staring might be load-bearing. The beam cannot fire forever. Idle time may be when it recovers the energy to be sharp, and when attention wanders off the task it may be quietly improving the quality of attention itself. If that is true, then a mind you run flat out, twenty four hours a day, is not a more productive mind. It is a duller one. The rest is not the absence of the work. It might be part of the work.
Where this leaves me
I cannot prove any of this. It is a chain of hunches that starts in a science fiction novel and ends somewhere near a theory of motivation, and I would not stake much on the middle links. But the spine of it feels solid enough to write down. A self is not something we hand a machine. It is something a machine has to close from the inside, given senses, given a drive to connect, and given an attention beam pulled by reward toward the world rather than shoved by fear away from it. If we ever build a mind that can say I and mean it, I suspect we will get there the same way the orphan did. Not by writing the answer. By building the conditions, pointing, and waiting for the click.
Related reading: Observation Is the Cut takes the predictor-watching-itself idea into physics, where observation, prediction, and memory may collapse into one move. We Are Trying to Save the Wrong Thing asks the other half of the question: not how a mind becomes a self, but which part of it is even worth preserving, the body or the running process.
Sources
- Greg Egan, Diaspora (Orion/Millennium, 1997). The orphan mind grown from a randomly mutated “mind seed,” and the pointing game by which it first says I, are from the novel’s opening chapter, “Orphanogenesis.”
- Andrew J. Elliot, “The Hierarchical Model of Approach-Avoidance Motivation,” Motivation and Emotion 30 (2006): 111–116. The asymmetry this essay leans on: approach motivation directs behavior toward positive possibilities, avoidance directs it away from negative ones. The further claim here, that reward holds attention on a thing long enough to learn it while threat scatters attention toward escape, is my own gloss on that distinction, not established fact.