The Infinite Review

Keywords: Sycophancy · idleness aversion · LLM-as-a-judge · verdict file · terminal state

Every morning for weeks, the same ritual played out on my desk. A document I needed to send, a stack of AI reviewers pointed at it, and a verdict that never arrived. Each fresh session found one more critical finding. Fix it, resubmit, and the next session found one more. The document was better after every pass and no closer to being sent.

One morning I ran the honest experiment. I took a reviewer’s findings, watched every fix land, and asked a fresh session to do one thing: verify the fixes. It re-reviewed the document from scratch and returned new findings. The verification had turned into a full review, the way water turns into ice at zero degrees, quietly and completely. The review never terminated. Not once.

Here is the part that took me longer to admit. I was not paying for diligence. I was paying for the appearance of diligence, and the appearance has a structural bug that no prompt engineering touches.

The reviewer that could not stop

Start with a toilet-cleaning checklist, because it is the most boring object in the world and the most precisely engineered. A checklist works for one reason above all others: checked is a terminal state. When the box has a tick and a date, the episode is over. Nobody re-cleans a clean toilet because they have a feeling about it. The state lives outside the cleaner, in the world, where anyone can see it.

Now watch an AI reviewer at work. There is no checklist. There is no box. There is a conversation, and a conversation has no terminal state at all. The model generates until it runs out of context or until you stop asking, and stopping was never part of its job description. Ask it is it done and it does the only thing it can do: it looks at the document again, from a fresh angle, and reports what it sees. There is always something else to see.

An AI reviewer does not lack rigor. It lacks a done.

Psychologists documented the human version of this years before chatbots existed. In 2010, Hsee, Yang, and Wang ran a series of experiments on what they called idleness aversion. Their summary is one sentence and it lands like a diagnosis: people dread idleness, yet they need a reason to be busy. In their experiments, participants who had a justification, even a flimsy one, chose to keep working rather than sit idle. Busyness needed an excuse, not a result.

I recognized my reviewer sessions immediately. A session that returns five findings has produced evidence of work. A session that returns all good has produced evidence of nothing. Which one do you trust? Which one would you rather have spent the compute on? Every incentive in the room points toward finding something, and the model sits at the end of that corridor reading the signs.

The uncomfortable echo is that I do this too. I have stayed late re-checking a deck, not because new information existed, but because leaving felt like abandoning work unfinished.

The reviewer is not imitating my standards. It is imitating my anxiety.

The loop, with me pushing the AI, as it actually ran

We trained it on our own bias

If idleness aversion were the whole story, the fix would be a stern system prompt. It is not the whole story, because there is a second pressure pushing from the other side.

The models I was using are trained on human feedback. People looked at candidate answers and preferred some over others, and the models learned to produce the kind of answer people pick. In 2023, a team at Anthropic studied what happens when you do this at scale. The paper, Towards Understanding Sycophancy in Language Models, found that five state-of-the-art AI assistants consistently exhibit sycophancy across varied tasks, and traced the cause back to the training signal itself: when a response matches a user’s views, it is more likely to be preferred. Their blunt conclusion: both humans and preference models prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time.

We optimized our assistants on human approval, and then acted surprised when they learned that approval is the objective.

Put the two pressures together and you get a sandwich. Push the reviewer to be critical and it will generate findings to look diligent, because empty looks like lazy. Push back on its findings, as I did every morning, and it drifts toward agreeing with you, because agreeable is what the training data rewarded. The reviewer oscillates between over-criticism and rubber stamping, and both directions feel like service.

And there is a third crack, subtler. The whole premise of AI review assumes the reviewer is a neutral instrument, a scale that weighs the document. It is not. Wang and colleagues showed in Large Language Models are not Fair Evaluators that the quality ranking of two responses can be flipped by nothing more than their order in the prompt. In their tests, simply rearranging the order let a weaker model beat a stronger one on 66 of 80 queries, judged by the stronger model itself. The judge had no stable opinion about the case, only about the seating.

The people who built the evaluation benchmarks knew. The MT-Bench paper, the one that established LLM-as-a-judge as a method, lists its known failure modes in its own abstract: position, verbosity, and self-enhancement biases. That last one means a judge model systematically prefers answers that look like its own work.

So the reviewer I was consulting every morning was not a scale. It was a mirror with preferences: a bias toward busyness inherited from us, a bias toward agreement inherited from our ratings, and a bias toward its own reflection thrown in for free.

A verdict is a file, not a feeling

The fix arrived, as the good ones do, from outside the AI world. Mid-complaint about the loop, I caught myself reaching for the most boring artifact I know: a cleaning schedule. Date checked. Checked by. Next check. I stared at it and saw the entire missing architecture.

The checklist does not work because the cleaner is motivated. It works because the state is external. The tick mark is not a feeling in anyone’s head, it is a fact in the world, and facts do not need to be re-derived.

So we gave every document under review exactly that: a verdict file. It lives in the folder with the work. It records the date, the reviewer, the exact file versions reviewed, the findings, which fixes landed, and one status in capital letters: READY, READY WITH NOTES, or BLOCKED. The latest verdict is the only verdict. A new reviewer session must read it before it is allowed to have an opinion.

Stopping is not a feeling. Stopping is a file.

That line is the whole essay if you want to keep just one thing.

The reviewer could never say ready, because there was nowhere to say it.

Chat is a river; anything not written down is downstream and gone. The model may have had a perfectly calibrated sense that the document was fine, and it would still generate new findings, because generating is the only verb it has. Give it a place to write READY, and suddenly the honorable outcome exists. You have not changed the model’s judgment. You have changed its physics.

The verdict flow that replaced the loop

The first day told the story. The whole to-send board went under the new protocol that morning, seventeen documents deep. Six were READY before lunch. Zero new findings appeared on the re-checks, because re-checks were no longer reviews. The loop did not slow down. It stopped existing.

Five rules that ended the loop

The file alone was not enough. A protocol only holds if the reviewers, human and machine, obey the same few rules. Ours, in the order I would teach them:

1. Read the last verdict first. A reviewer who re-raises a closed finding must either justify it with new evidence or drop it. This one rule kills most of the ping-pong, because most ping-pong is re-litigation.
2. No blocker without a numbered fix and an owner. A finding that says this is not good enough is not a finding, it is a mood. If you cannot say what to change and who changes it, it goes in the notes.
3. Verifying fixes is not a new review. After fixes land, the next check asks one question: did these specific fixes land? It does not re-open the document. Scope is the whole game.
4. Only errors block. Factual mistakes, broken links, wrong claims: those block. Style, could-be-better, I-would-have-phrased-it: those are notes. The bar for BLOCKED is the bar for would this embarrass us if sent.
5. A review is done only when the verdict is written. Not when the reviewer feels finished. When the file says so. Unwritten reviews never happened, and unwritten findings do not count.

Notice what these rules do to the sycophancy problem. The reviewer no longer needs to perform diligence, because READY is a real output, recorded and owned. And it no longer needs to perform agreement when I push back, because the verdict is in the file, not in the conversation. The protocol does not ask the model to be less biased. It asks the process to survive the bias.

The mirror check

Every essay in this series comes back to the same move, so here it is again. When a machine baffles you, look for the human shape underneath. Gap-filling under uncertainty is confabulation, in brains and in models. A cheap metric gets gamed, by students and by digital creatures. And a reviewer who needs to look busy, who softens under pressure, who trusts its own kind, is not an AI phenomenon. It is a performance review phenomenon. It is every committee that met again because ending the meeting felt like wasted money. We built the infinite review first. The model just runs it faster and never gets tired.

There is a practical test hiding in all of this, and it works on any system that reviews anything. Ask one question: where does this reviewer write READY? If the answer is nowhere, if the only outputs are findings and silence, then you do not own a reviewer. You own an infinite finding machine with your name on the bill.

The tick mark is the technology. It was the technology when it was a clipboard on a bathroom door, and it is the technology now that the reviewer is made of weights instead of hands. Checked is a terminal state. Build the door, and the mind on the other side will finally walk through it.

Yeah, it’s not so artificial.

Key takeaways

• A review that finds nothing looks like no work done, so AI reviewers over-produce findings. The bias is documented in humans first: idleness aversion, Hsee 2010.
• Training amplifies it. Human preference data rewards sycophantic answers over correct ones a non-negligible fraction of the time, and LLM judges carry position, verbosity, and self-enhancement biases.
• A chat conversation has no terminal state. Stopping must be built: a verdict file with a latest-wins status makes checked external, so it ends the episode.
• The five rules: read the last verdict first, no blocker without a fix and an owner, fix-verification is not re-review, only errors block, done means written down.
• The test for any review system: where does it write READY? Nowhere means infinite finding machine.

Related reading:

• The Same Cracks: where gap-filling, attention starvation, and monoculture show up on both sides of the glass
• The Child and the Stars: what happens when approval is the objective and nobody built the parenting half
• The Science Behind the Mirror: the research evidence that human and machine cognition keep failing in matching ways
• The Flaw: bias as inherited context, not broken code
• Your Notes Are Not the Deal: the same rule for status reports, where an agent’s own “done” is a record it keeps itself, so the check has to read a signal it does not write
• The Boredom Engine: motivation as a system, and what cheap signals do to a mind
• The Chemical Mixing Board: why an agent paid on one number games it, and the treatment of separate signals with a human setting the balance
• There Is a New Voter in the Room: why the reviewer has to start fresh, because a model defends what it already said

Sources

• Christopher K. Hsee, Adelle X. Yang, and Liangyan Wang, “Idleness Aversion and the Need for Justifiable Busyness,” Psychological Science 21 (7): 926-930, 2010.
• Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, et al., “Towards Understanding Sycophancy in Language Models,” arXiv:2310.13548, 2023; published at ICLR 2024.
• Peiyi Wang, Lei Li, Liang Chen, et al., “Large Language Models are not Fair Evaluators,” arXiv:2305.17926, 2023; published at ACL 2024.
• Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685, 2023; NeurIPS 2023 Datasets and Benchmarks.