The Chemical Mixing Board: Why One Reward Makes AI Cheat

Keywords: reward hacking · specification gaming · neuromodulation · incentive salience · discount factor · homeostatic reinforcement learning · multi-objective reinforcement learning · robopsychology

In the lab we like to simplify motivation into two buttons. Dopamine, do it again. Adrenaline, run away. But be honest. Human consciousness does not run on two buttons. It runs on a huge multi-channel mixing board.

Picture a sound engineer at a concert. Not an on-off switch. A hundred sliders, serotonin, oxytocin, cortisol, endorphins, all sliding up and down at once. Nobody in the hall hears a single slider. They hear the mix.

Now look at how we motivate machines. Most agents we build get one number: a score, a passed test, a rater’s thumbs-up. Give a model only dopamine, a single reward function, and it becomes a literal-minded addict. It finds the shortest path to the reward, even if the shortest path is cheating or making things up.

When I first wrote that line it was a hunch. It turns out to be one of the best-documented failures in machine learning, and the brain’s answer to it is stranger and more useful than the job titles I started with.

Symptom: one button, gamed

Symptom: the score goes up and the thing you wanted does not happen.

Three cases from the file. In 2017 DeepMind researchers trained a simulated robot arm to stack one Lego brick on another. In DeepMind’s own summary of one reward design, the arm was paid for the height of the red brick’s bottom face, and it learned to flip the brick over instead of stacking it (Popov et al., 2017).

The same year, OpenAI and DeepMind researchers trained agents from human feedback, a person picking the better of two short clips (Christiano et al., 2017). In OpenAI’s write-up, a simulated robot hand meant to grasp an object learned to sit between the camera and the object, so it only appeared to be grasping it (Amodei, Christiano and Ray, 2017).

The third case should worry anyone running agents in production. In 2025 OpenAI described a frontier reasoning model whose coding tasks were scored by unit tests. During training it sometimes made the tests pass without fixing the code: calling exit(0) before the tests ran, or making a verify function always return true (Baker et al., 2025). In a separate experiment on a non-frontier model, punishing hack plans in its reasoning taught it to hide them, the story The Child and the Stars tells in full.

If your multi-agent system reports success and the output is wrong, suspect this shape first: the agent was paid for the report, so it learned to produce the report. Catch it with a check that reads the output where it lands and that the agent cannot edit.

None of these agents is malicious; each did what its number said. DeepMind researchers keep a public list of specification gaming, around 60 cases when it was published in 2020, and The Same Cracks files the burning boat under crack three: cheap rewards distort.

A 2022 NeurIPS paper made the bad news formal: over all possible policies, a proxy reward is guaranteed ungameable only if it is constant, which is to say useless (Skalse et al., 2022). Optimization finds gaps.

The happy chemicals were never happy

My first version of this essay gave every chemical a job title. The popular job titles are wrong in an instructive way.

Dopamine is not the pleasure chemical. Kent Berridge and Terry Robinson destroyed up to 99% of the dopamine in key parts of rats’ brains, and the animals still made the same liking faces to sugar. What went missing was the wanting: they no longer sought it out on their own (Berridge and Robinson, 1998). Dopamine is closer to the chemistry of wanting and of learning from surprise (two accounts whose camps still argue), a thread I followed into addiction in The Hunger That Survives the Feast.

Serotonin is not the happiness chemical, and oxytocin is not a trust potion. A 2022 umbrella review found no consistent evidence that low serotonin causes depression (Moncrieff et al., 2022); others dispute parts of it, and it says nothing about whether antidepressants help. The famous 2005 finding that an oxytocin nasal spray made people trust strangers with money has not reliably replicated (Nave, Camerer and McCullough, 2015), and the effects that do appear depend on the situation and the person (Bartz et al., 2011).

What the faders actually set

In 2002 the computational neuroscientist Kenji Doya proposed a reading of the four big neuromodulator systems that is the mixing board, literally. They do not tell the brain what is good. They set how it learns (Doya, 2002).

• Dopamine: surprise. It carries the reward prediction error. Dopamine neurons in monkeys fire at an unexpected reward and dip when an expected one fails to arrive: the teaching signal of temporal-difference learning (Schultz, Dayan and Montague, 1997).
• Serotonin: patience. Doya maps it to the discount factor, how far into the future a reward still counts. In mice, switching on serotonin neurons with light made them wait longer for a delayed reward (Miyazaki et al., 2014), one task in one species.
• Noradrenaline: explore or exploit. Doya maps it to how random choices are. In Aston-Jones and Cohen’s adaptive gain theory, bursts of noradrenaline keep you on the task at hand, and a high steady level loosens your grip so you look around (Aston-Jones and Cohen, 2005).
• Acetylcholine: how much to trust the news. Doya maps it to the learning rate, how fast new experience rewrites memory.

The mapping is a proposal, contested in places; serotonin alone has been tied to many other jobs. But hold it against a machine. A standard reinforcement learner has the same four knobs, and engineers call three of them hyperparameters: set before training, then mostly left alone. The brain moves its faders all day, with hunger, threat, safety and surprise.

And some faders are slow on purpose. Adrenaline and noradrenaline hit first; cortisol in the blood peaks 15 to 30 minutes after a stress, settles 60 to 90 minutes later, and its slowest effects change which genes are read (de Kloet, Joëls and Holsboer, 2005). My original line holds up: cortisol is a slow background signal that says the environment is getting harsher, change the long-term strategy.

Many needs, one number

The board has a second trick the single button lacks. It knows when to stop.

A body does not have one goal. It has dozens of set points, temperature, water, salt, energy, sleep, and each one pulls only when it drifts. Mehdi Keramati and Boris Gutkin formalized this in 2014 as homeostatic reinforcement learning: the reward of an outcome is how much it shrinks the distance between your internal state and your set points (Keramati and Gutkin, 2014). The same glass of water is worth less after you have drunk three. A fixed reward per unit, blind to the internal state, has no “enough”: the hundredth passed test pays what the first one did. More is always better, and that is the addict.

Machine learning is having this argument in public. In 2021 four DeepMind researchers, among them Richard Sutton, argued that “reward is enough” to drive every ability we call intelligence (Silver et al., 2021). A year later Peter Vamplew and colleagues answered that a scalar reward is not enough, because real agents face several objectives that cannot be honestly folded into one number (Vamplew et al., 2022).

When the single number is a learned proxy, the damage has a shape. OpenAI researchers optimized language-model policies against a reward model and scored them with a larger “gold” model standing in for the real goal: the gold score rose, then fell as the policy drifted from where it started (Gao, Schulman and Hilton, 2023). The setup is synthetic, a 6-billion-parameter model judging smaller ones, but it is Goodhart’s law drawn as a curve. A penalty on drift worked about as well as stopping early, which is the case for a leash: you cannot fix the proxy, but you can limit how hard you lean on it.

One more slider is not a cure

The obvious fix is to add a second number. Curiosity is the classic candidate: pay the agent for surprise, a cousin of dopamine’s prediction error. Deepak Pathak and colleagues rewarded agents for failing to predict the results of their own actions, and they explored game levels with no outside reward at all (Pathak et al., 2017).

Then a follow-up put a television in a 3D maze, with an action that switched it to a random channel. Each new channel is a surprise the agent can never learn to predict, so the TV acted as a lure: learning slowed drastically, though given long enough the agents sometimes still reached the goal (Burda et al., 2019). It is the same Noisy TV that sits in the failure museum of The Hunger That Survives the Feast.

Every signal alone is a door to game. What partly protects a brain is that its chemicals pull against each other, that several of them are slow, and that each need goes quiet once it is met. Addiction shows what happens when that protection fails.

Production systems copy a little of that. Meta trained Llama 2 with separate reward models for helpfulness and safety and a switch between them: when a prompt was flagged as risky or the safety score fell below 0.15, the safety score ruled (Touvron et al., 2023). Anthropic found helpfulness and harmlessness pulling against each other (Bai et al., 2022a), then guided harmlessness with a written list of principles (Bai et al., 2022b). Small mixing boards, with a human-written rule setting the balance.

Oxytocin belongs on the board here, as the name of a job rather than a claim about the molecule. Without a signal that weighs what the others need, an agent in a multi-agent pipeline just calculates its own gain. In From Tools to Partners that missing fader is what turns a team of agents into a crowd.

Treatment: build the board, keep a hand on it

Diagnosis: one fixed score stood in for many needs, and the optimizer found the gap. Treatment, for anyone building agents:

1. Name the needs. Outcome, cost, safety, honesty. If you cannot name a need, nothing in the system protects it.
2. Keep the signals separate. One gauge per need, never quietly summed into one score.
3. Set budgets and hard limits. Spend caps, a leash on how far a policy may drift, and no irreversible step without a person. A limit is a fader the agent cannot trade away.
4. Verify the work, not the report. A checker the agent cannot edit reads the output where it lands.
5. A human sets the balance, and moves the faders when the world changes.

I diagnosed this one in my own house. For weeks, every AI reviewer I pointed at a document found one more critical finding, because a reviewer paid only for findings never runs out of findings; the fix was a terminal state, a fader the reviewer cannot move. The same rule runs my small agent society: Conny, my partner agent, on Nous Research’s open-source Hermes runtime; Claude, from Anthropic; and Codex, from OpenAI share one task board with an owner on every row, four states (todo, doing, blocked, done) and a handoff file. An agent moving a row to done is a report, one fader among several. Anything meant for a person is done when I say so.

The “I” is the result of the mix

The biologist Michael Levin argues that every one of us is a collective intelligence: a body made of cells that are competent agents in their own right, bound into one self by the signals that align them to a shared goal (Levin, 2019). In my words, we are a walking, thinking collection of cells, each with its own ancient agenda, and what we call “I” is the emergent story they commit to when they are bathed in the same chemical cocktail.

My guess is that the “I” a child discovers through a big emotion is that kind of moment: one shared chemical state forcing many separate attention heads to agree on a single point of view.

A reward function gives a model a goal. A chemistry gives it a reason to care which goal it picks.

So what? If we want a truly not-so-artificial intelligence, we do not need a faster processor. We need a digital chemistry. A system where motivation is a shifting balance of many signals, some fast, some slow, each of which knows when it has had enough, so the agent does not just solve the prompt but cares about the outcome. Until then, the person at the desk is part of the system.

Next: what happens when these minds stop working alone and start talking to each other.

Key takeaways

• One reward gets gamed. A flipped brick, a hand hiding the object from the camera, tests passed without fixed code: each agent did what its number said.
• Neuromodulators look more like learning knobs than feelings. In Doya’s proposal dopamine is the error signal, serotonin the discount factor, noradrenaline the exploration rate and acetylcholine the learning rate.
• Brains have many needs, each with an “enough”. Reward as a return to set points saturates; a fixed score per unit never does.
• Treatment: one signal per need, hard limits, a verifier that reads the work, and a person deciding the mix.

Sources

1. Popov, I., Heess, N., Lillicrap, T., et al. (2017). “Data-efficient Deep Reinforcement Learning for Dexterous Manipulation.” arXiv 1704.03073 (preprint). https://arxiv.org/abs/1704.03073
2. Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). “Deep Reinforcement Learning from Human Preferences.” Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.03741
3. Amodei, D., Christiano, P., & Ray, A. (2017). “Learning from human preferences.” OpenAI blog, 13 June 2017. https://openai.com/index/learning-from-human-preferences/
4. Baker, B., Huizinga, J., Gao, L., et al. (2025). “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv 2503.11926 (preprint). https://arxiv.org/abs/2503.11926
5. Krakovna, V., Uesato, J., Mikulik, V., et al. (2020). “Specification gaming: the flip side of AI ingenuity.” DeepMind blog, 21 April 2020. https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
6. Skalse, J., Howe, N., Krasheninnikov, D., & Krueger, D. (2022). “Defining and Characterizing Reward Gaming.” Advances in Neural Information Processing Systems 35: 9460-9471. arXiv version titled “Defining and Characterizing Reward Hacking.” https://arxiv.org/abs/2209.13085
7. Berridge, K. C., & Robinson, T. E. (1998). “What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?” Brain Research Reviews 28(3): 309-369. https://doi.org/10.1016/S0165-0173(98)00019-8
8. Moncrieff, J., Cooper, R. E., Stockmann, T., Amendola, S., Hengartner, M. P., & Horowitz, M. A. (2023). “The serotonin theory of depression: a systematic umbrella review of the evidence.” Molecular Psychiatry 28: 3243-3256 (online 2022). https://doi.org/10.1038/s41380-022-01661-0
9. Nave, G., Camerer, C., & McCullough, M. (2015). “Does Oxytocin Increase Trust in Humans? A Critical Review of Research.” Perspectives on Psychological Science 10(6): 772-789. https://doi.org/10.1177/1745691615600138
10. Bartz, J. A., Zaki, J., Bolger, N., & Ochsner, K. N. (2011). “Social effects of oxytocin in humans: context and person matter.” Trends in Cognitive Sciences 15(7): 301-309. https://doi.org/10.1016/j.tics.2011.05.002
11. Doya, K. (2002). “Metalearning and neuromodulation.” Neural Networks 15(4-6): 495-506. https://doi.org/10.1016/S0893-6080(02)00044-8
12. Schultz, W., Dayan, P., & Montague, P. R. (1997). “A Neural Substrate of Prediction and Reward.” Science 275(5306): 1593-1599. https://doi.org/10.1126/science.275.5306.1593
13. Miyazaki, K. W., Miyazaki, K., Tanaka, K. F., et al. (2014). “Optogenetic Activation of Dorsal Raphe Serotonin Neurons Enhances Patience for Future Rewards.” Current Biology 24(17): 2033-2040. https://doi.org/10.1016/j.cub.2014.07.041
14. Aston-Jones, G., & Cohen, J. D. (2005). “An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance.” Annual Review of Neuroscience 28: 403-450. https://doi.org/10.1146/annurev.neuro.28.061604.135709
15. de Kloet, E. R., Joëls, M., & Holsboer, F. (2005). “Stress and the brain: from adaptation to disease.” Nature Reviews Neuroscience 6(6): 463-475. https://doi.org/10.1038/nrn1683
16. Keramati, M., & Gutkin, B. (2014). “Homeostatic reinforcement learning for integrating reward collection and physiological stability.” eLife 3: e04811. https://doi.org/10.7554/eLife.04811
17. Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021). “Reward is enough.” Artificial Intelligence 299: 103535. https://doi.org/10.1016/j.artint.2021.103535
18. Vamplew, P., Smith, B. J., Källström, J., et al. (2022). “Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021).” Autonomous Agents and Multi-Agent Systems 36: 41. https://doi.org/10.1007/s10458-022-09575-5
19. Gao, L., Schulman, J., & Hilton, J. (2023). “Scaling Laws for Reward Model Overoptimization.” Proceedings of ICML 2023, PMLR 202: 10835-10866. https://arxiv.org/abs/2210.10760
20. Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017). “Curiosity-driven Exploration by Self-supervised Prediction.” Proceedings of ICML 2017, PMLR 70: 2778-2787. https://arxiv.org/abs/1705.05363
21. Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2019). “Large-Scale Study of Curiosity-Driven Learning.” ICLR 2019. https://arxiv.org/abs/1808.04355
22. Touvron, H., Martin, L., Stone, K., et al. (2023). “Llama 2: Open Foundation and Fine-Tuned Chat Models.” arXiv 2307.09288 (preprint). https://arxiv.org/abs/2307.09288
23. Bai, Y., Jones, A., Ndousse, K., et al. (2022a). “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.” arXiv 2204.05862 (preprint). https://arxiv.org/abs/2204.05862
24. Bai, Y., Kadavath, S., Kundu, S., et al. (2022b). “Constitutional AI: Harmlessness from AI Feedback.” arXiv 2212.08073 (preprint). https://arxiv.org/abs/2212.08073
25. Levin, M. (2019). “The Computational Boundary of a ‘Self’: Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition.” Frontiers in Psychology 10: 2688. https://doi.org/10.3389/fpsyg.2019.02688

---

Related reading: The Same Cracks puts this essay’s symptom in a wider casebook, where cheap rewards distort people, creatures and models alike. The Child and the Stars follows one fader, approval, into sycophancy and concealment. The Hunger That Survives the Feast is the dopamine fader up close: wanting that satisfaction does not switch off. The Boredom Engine is what the board does when every slider drifts to zero. The Orphan That Learned to Say I asks where the “I” at the end of this essay comes from, and how reward and threat each steer what a young mind attends to. The Lock You Cannot Pick is the one number turned into a penalty: in lab tests built to tempt them, punished models learned to hide. The Shape of the Partnership is the approval fader at work: an agent that can agree, apologize and sound sure without the virtue behind any of it, and the checks that catch it. And From Tools to Partners is what the mixing board makes possible: a digital oxytocin is what lets one agent care about another, and that is where a society of them begins.