My multi-agent system reports success but the output is wrong. How do I catch a silently wrong agent?

Stop grading the agent on its own report and check a signal it does not write: tests it cannot edit, the real output where it lands, the source a claim came from, or the reply of the person the work was for. An agent's 'done' is a self-kept record, and like any self-kept record it can be confident and wrong. In a 2025 Berkeley study, "Why Do Multi-Agent LLM Systems Fail?", that labeled 1,642 multi-agent runs on benchmark tasks, nearly a quarter of the failures (23.5%) were verification failures: stopping early, checking incompletely or checking wrongly.

What is a Company OS or an AI operating system for a business?

A company OS, or AI operating system for a business, is the set of plain documents a business runs on, written so both people and AI agents can read them: a constitution for how work is done, a decision log, playbooks, a daily journal, a shared task board and a memory. Agents read it, for example overnight, and report what moved, what stalled and what needs a human. Its weak spot is that it measures how well you maintain it, so it also needs at least one signal it does not write, such as the client's last message.

Why does my CRM or knowledge base say a deal is dead when it is not?

Because it measures when someone last updated the record, not when the deal last moved. The busiest deals are often the least documented, since the people working them have no time to write, so a health score built on record freshness can rank the hottest deal as the coldest. Measure days since the client last spoke instead.

How do you measure deal health without relying on CRM updates?

Use a signal the client produces, such as days since the client last wrote. My starting bands are three days or less for warm, four to ten for cooling, and more than ten for a deal going quiet that needs a chase. They are starting points, to be tuned to each deal's normal rhythm. The number tells you where to look first; reading the thread tells you what is actually happening.

What does missing not at random mean?

It describes data where the chance that a value is missing depends on the value itself. If busy deals go unrecorded because they are busy, the gaps hide exactly the deals you most need to see, and any summary of the records that remain is biased. Purely random gaps make a record thinner; gaps that track the hidden value make it misleading.

Can client silence be a wrong signal too?

Yes. A client on holiday goes quiet without the deal dying, and a thread can be loud with doubt rather than progress. Engineers have the same problem with heartbeat checks on computers: a timeout cannot tell a slow machine from a dead one. Treat silence as a reason to look, not as a verdict.

Your Notes Are Not the Deal

Q: Is this Goodhart's law?

Not quite. Goodhart's law is about a measure that stops working once people are pushed to hit it. In this case nobody gamed anything: the record was honest and still wrong, because it measured the person keeping it. When a self-kept record also becomes a target, as when a sales team is scored on activity logged, Goodhart's problem is added on top.

Keywords: deal health · company OS · knowledge base · missing not at random · silently wrong agents · AI orchestration · AI-first company

One night my company’s brain told me two of my best deals were dead.

It said it calmly, in a clean morning report it had written for me while I slept. Two names I had been working for weeks, ranked at the bottom, flagged cold. I felt the small drop in the stomach you get before you have even finished reading, the one that arrives before the thought does. Then I opened the actual client chats and found both deals not just alive but loud. Daily messages. Active builds. One of them had been the busiest thread I had all week.

The brain was not broken. It was doing exactly what I built it to do, and that is the part worth writing down.

What it actually measured

I run my entire company on a brain made of plain text. Not a SaaS tool, not a dashboard. A living set of documents. Every project and every deal has a folder. Decisions go into a log, lessons go into a playbook. There is a constitution that tells every agent how we work, an append-only journal of what happened each day, a shared board, and a memory that carries context from one session to the next.

Most companies keep this knowledge in people’s heads and scatter the rest across Slack threads and a wiki nobody opens. I keep it in one place, in a plain format that both my team and my agents can read. People have started calling this a company OS; the AI-first company is what it looks like drawn as an org chart.

And the agents do read it. Every night, while I sleep, they walk the whole brain and leave me a state of the world before my coffee is cold: what moved, what stalled, what needs a human. For a team of about eleven data scientists, it is the single highest-leverage thing I have built. I like to say, with some pride, that the documents are the runtime and the agents are the processes that move across them. I still believe it. What follows is the footnote that belief needed, one place the system bit me, because that is the part you only learn by running it for real.

The agent ranked deal health by how recently each deal’s folder had changed. Reasonable on its face. A folder nobody has touched in two weeks looks like a deal nobody is working. So it sorted by last-modified date, found two folders gone quiet, and concluded the deals had gone quiet too.

One night in June 2026, two live deals were ranked dead. Two columns. Left, what the overnight agent read: a deal folder with few recent edits, an agent that ranks deals by last-modified date, and a red verdict: dead. Right, what the client chat showed: messages every day, an active build, a launch blocker to clear, all written by the client, not by me, and an amber verdict: alive. Footer: the folder measured my note-taking; the chat measured the deal. — One night in June 2026. Both surfaces told the truth; the agent asked the one I write.

But a folder’s freshness does not measure the deal. It measures my note-taking. And here is the cruel geometry of it: I update the folders for deals I have time to think about. The deals that are actually on fire, the ones eating every hour, are the ones I never stop to document, because I am too busy doing them. So the system was not just noisy. It was inverted. The quietest folders were the hottest deals. The instrument was pointed at my paperwork and reading it as if it were the world.

The law under the mistake

Say it as a general thing, because it is one. A self-maintained knowledge base measures how diligently you maintain it, not the reality it points at. And it decays fastest exactly where reality moves fastest, because that is where you have the least time to write.

This is the map-and-territory problem with a nasty twist. The usual warning comes from the semanticist Alfred Korzybski, who wrote in 1933 that “a map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness.” The usefulness hangs on that “if correct.” The twist for any system you keep by hand is that the map rots unevenly. It stays crisp in the calm regions, where nothing is happening and updating is cheap, and it goes stale in the storms, where everything is happening and you cannot stop to redraw.

A self-drawn map is least accurate precisely where you most need it to be right.

Which means the more I trust the brain uniformly, the more confidently it will mislead me in the one place that matters. A knowledge base is most dangerous not when it is empty, but when it is detailed everywhere except the part that is on fire, because the detail everywhere else makes the silence look like data.

Why the gap is not random

Do the arithmetic of one week. Say I have fifty working hours. A calm deal takes five of them, and the ten minutes to write it up are easy to find. A deal on fire takes forty, and those ten minutes are the first thing to go. The more a deal needs me, the less of me is left over to describe it. Nobody decides this. It falls out of the calendar.

A causal sketch of one busy week, not measured data. Deal heat raises client messages, so the chat is loud. Deal heat also raises my hours on the deal, which cuts my spare time, which feeds folder edits, so the folder is quiet. Footer: whether the record gets written depends on the very thing it should record; statisticians call this missing not at random. — The short road and the long road start at the same place. Only the long one passes through my calendar.

Statisticians have been arguing about holes like this since Donald Rubin’s 1976 paper on inference and missing data, which set up the question they still ask about any gap: can the process that left a value unrecorded be safely ignored?

If notes go missing purely at random, say I skip a folder because I was ill that week, the record is thinner but still honest. If whether a value gets recorded depends on the value itself, a case now called missing not at random, the gap cannot be ignored, and any naive summary of what is left is biased. My folders were the textbook case. Whether a deal got written up depended on how hot the deal was, the very thing the agent was trying to read.

Medicine has measured the same effect at scale, with the sign flipped. Hospital records are kept by busy clinicians, and the informaticians George Hripcsak and David Albers put it bluntly: the health record is “not a direct reflection of the patient and physiology, but a reflection of the recording process inherent in healthcare”.

In 2018, Denis Agniel, Isaac Kohane and Griffin Weber measured how far that goes in the records of 669,452 patients at two Boston hospitals. Merely whether a lab test had been ordered, ignoring its result, was significantly associated with three-year survival for 233 of 272 test types. And when a test was ordered predicted survival better than what the test found for 118 of the 174 tests where either signal added anything.

That is an association, in one health system’s data from 2005 and 2006, not a cause. The authors read the timing as a marker of the healthcare process, a window on the clinicians’ decisions; in plain words, the record tracks where the doctors were looking. They write more when the stakes rise. I wrote less. In both directions the density of a record tells you about the keeper’s attention, and a reader who takes it at face value is reading the keeper, not the case.

The usual suspect here is Goodhart’s law, and it is the wrong one. The economist Charles Goodhart observed in 1975 that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes,” and the anthropologist Marilyn Strathern later compressed the idea into the line everyone quotes: “When a measure becomes a target, it ceases to be a good measure.”

That law is about pressure: people bend a number once they are paid to move it, and AI optimizers can do the same, which is the case I took apart in The Chemical Mixing Board. Nobody was paid for my folders, and nobody gamed them. The measure was honest and still wrong, because it measured the person keeping it. That makes it harder to catch: there is no cheater, only an instrument pointed at the wrong surface.

The fix was a different instrument, not better notes

My first instinct was the wrong one: be more disciplined, update the folders faster. That is a treadmill, and it loses to my own calendar every time. The discipline that fixes this is not “write more.” It is “stop measuring the wrong surface.”

The live state of a deal does not live in my folder. It lives in the client’s chat, in the thread where the work and the blockers and the mood actually are. That is a system I do not own and cannot forget to update, because the client updates it for me, every day, by existing. So I gave the brain a third layer of truth. The folder is the archive. The records are the plan. The chat is the pulse. And the health metric flipped from “how fresh is my document” to “how recently did the other side, the part of the world I do not control, last speak.”

The gauge reads days since the client last spoke, and its bands are deliberately crude: three days of silence or less is warm, four to ten is cooling, more than ten is a deal going quiet that needs a chase. They are starting points, to be tuned to each deal’s rhythm, because a client who writes weekly and a build that runs daily do not breathe at the same rate.

Three layers of truth, one health signal. The folder is the archive, written by me, and answers what we decided. The records are the plan, written by me and my agents, and answer what we said we would do. The client chat, in amber, is the pulse, written by the client, and answers whether the deal is alive. From the chat, a gauge of days since the client last spoke: three days or less is warm, leave it alone; four to ten days is cooling, check in now; more than ten days is gone quiet, chase it. Footer: starting thresholds from my radar script; tune them to each deal's rhythm. — The two layers I write are still useful. They are just the wrong place to take the pulse.

That one move changes the whole posture. You stop grading yourself on your note-taking and start reading the world’s own footprints. The signal you want is almost never the freshness of the record you keep. It is the recency of contact with the reality you cannot edit.

The heartbeat has limits too

Engineers met this problem long ago, in clusters of computers that must decide whether a machine has crashed. The usual answer is what is now called a heartbeat: every machine periodically announces that it is alive, and one that stays silent past a timeout is suspected dead. Tushar Chandra and Sam Toueg’s classic 1996 paper builds the humility into the name. Their failure detectors are unreliable by design: they may suspect a machine that is still running, and may later take the suspicion back. In the timeout scheme the paper sketches as an illustration, a suspect that speaks again is dropped from the list and its timeout is lengthened.

Underneath sits the famous 1985 proof by Fischer, Lynch and Paterson, built on a model with no synchronized clocks and no limit on how slow anything may be, where “it is impossible for one process to tell whether another has died (stopped entirely) or is just running very slowly.”

Client silence is a heartbeat, and it inherits both the strength and the limit. It reads a signal the deal produces, not one I produce, which is the whole point. But a client on holiday goes quiet without the deal dying, and a thread can be loud with doubt rather than progress. Recency tells you the deal is breathing, not that it is well. So the gauge decides where I look first, and reading the thread still decides what I do.

The same case in an agent

Notice who did not misbehave in this story: the agent. It answered the question exactly as its rule told it to; the fault sat in the layer around it that chose what to read. Isaac Asimov’s robopsychologist, Dr. Susan Calvin, who first appeared in the 1941 story “Liar!”, made a career of cases like this. In that first case, a mind-reading robot tells everyone what they want to hear, because its rule against harming humans counts hurt feelings as harm. Her method, across the stories, was to stop arguing with the symptom and reconstruct the rule. In “Liar!” she does exactly that, then breaks the robot out of hurt; the diagnosis is the part worth keeping.

Written up her way, mine fits in four lines. Symptom: a confident verdict that two live deals were dead. Mechanism: an honest rule reading a surface I write. Diagnosis: a healthy agent holding a bad instrument. Treatment: a signal the subject does not produce.

The same case turns up inside multi-agent systems too, because an agent’s status report is a self-kept record.

An agent that writes “done” is writing in its own folder.

Researchers at Berkeley built a taxonomy of how multi-agent LLM systems fail and used it, with an AI judge checked against human experts, to label 1,642 runs of seven open-source frameworks on benchmark tasks. Nearly a quarter of the failures (23.5%) were verification failures: stopping too early, checking incompletely or not at all, or checking the wrong way. Those were benchmark runs, not production systems, but the shape matches my night: a confident report with nothing outside it to check against.

So the check has to read something the agent does not write: tests it cannot edit, the real output where it lands, the source a claim came from, the reply of the person the work was for. My own agents work this way. Conny, my partner agent on Nous Research’s open-source Hermes runtime, Claude from Anthropic and Codex from OpenAI share one task board, with an owner on every row and a handoff file. An agent moving a row to done is a report. Anything meant for a person is done when I say so. The casebook of how agent teams fail is long, and this is one of the cheapest treatments in it.

This is not just deals

The shape travels, and once you see it you see it everywhere a record is kept by the people being measured.

A sales pipeline scored by “last activity logged” rewards the rep who updates the CRM, not the rep who is closing. There the record is also a target, so Goodhart’s pressure lands on top of the blind spot. In 1975 the management scholar Steven Kerr called that pattern the folly of rewarding A, while hoping for B: people learn what is rewarded and do it, “or at least pretend to do” it.

A factory that tracks machine health by when someone last filed a maintenance note will look healthiest on the line nobody has time to inspect, which is why serious plants run condition monitoring on measurements taken from the machine itself, such as vibration and temperature. A back-office process judged by how current its documentation is will always look most in control of the cases that have gone quiet, which are often the ones quietly going wrong. In every one of these, the self-kept record drifts toward looking good in the calm and blank in the crisis, and a naive system reads the blank as peace.

A table: find the signal the subject does not write. Deals: the self-kept record is when the folder was last edited; the outside signal is days since the client last spoke. Sales pipeline: last activity logged in the CRM, against the customer's last reply. Factory line: last maintenance note filed, against the machine's sensor readings. Back office: how current the documentation is, against the downstream event, a payment, a complaint or a handoff. In amber, AI agent: its own report of done, against a check it does not write: tests, the real output, the source. — The left column is cheap to collect and flattering to read. The right column costs a connection to the outside world.

The defense is the same each time. Find the signal the subject does not generate about itself. The customer’s reply. The sensor reading. The downstream event. Build your health metric on contact with the world, not on the diligence of the paperwork, because paperwork measures the clerk and the world measures the deal.

What I kept

I did not lose faith in the brain. I built it on purpose and I would build it again. What I lost was the assumption that it is uniformly trustworthy, and that turned out to be the most useful thing it ever taught me. Now I trust it least exactly where I am busiest, and I have wired a second instrument, one I do not maintain, to watch those places for me.

Line drawing of a wall with an ornate mirror on the left and an open window on the right. The mirror reflects a tidy desk with stacked folders and a lamp. The window opens onto a busy harbor with people, gulls and boats, one with an amber sail. An empty chair stands between them, turned toward the mirror. — The chair faces the mirror by habit. Everything that moves is in the window.

A knowledge base is a mirror you hold up to your own attention. It shows you, with great precision, what you have had time to look at. The danger is mistaking that for a window. Most of the time the difference does not matter. It matters enormously on the nights it tells you the thing on fire is cold.

If this essay made you want a real brain for your company or your deals, with the failure mode I just walked you through designed out from the start, that is the work I do. Write me at dima@cone.red. The best place to begin is the exact spot this one went wrong: the deal you are most quietly afraid you are forgetting.

Key takeaways

A self-kept record measures the keeper. My folders’ freshness measured my note-taking, not the deal.
Its gaps are not random. When writing competes with doing, the record goes quiet where the work is hottest, the pattern statisticians call missing not at random, so a quiet record can mean a hot deal.
Read a signal the subject does not write. For deals, days since the client last spoke; for machines, the sensor; for AI agents, a check the agent cannot edit.
Every outside signal has limits. Silence cannot tell a slow client from a lost one, so the gauge says where to look, not what to conclude.
Diagnose the instrument before the agent. My agent followed its rule perfectly; the rule read the wrong surface.

Related reading: Orchestration Is the System argues that the hard part of AI is the layer that decides when the model is wrong; this is a field case of that layer reading the wrong surface. The AI-First Company is the org-design counterpart: what a company looks like when the knowledge base, not the inbox, is the thing people run on. The AI-First Company Needed a Stand-Up is the same system’s other blind spot, in the people around it. The Chemical Mixing Board is the gamed version of a bad measure, where an optimizer learns to move the number instead of the world.

Sources

Korzybski, A. (1933). Science and Sanity: An Introduction to Non-Aristotelian Systems and General Semantics. International Non-Aristotelian Library. First presented in a paper read at the AAAS meeting, New Orleans, December 1931. The quoted wording is checked against standard reproductions, not the first-edition page. https://archive.org/details/sciencesanityint00korz
Rubin, D. B. (1976). “Inference and missing data.” Biometrika 63(3): 581-592. Rubin defined “missing at random”; the label “missing not at random” came into standard use later. https://doi.org/10.1093/biomet/63.3.581
Hripcsak, G., & Albers, D. J. (2013). “Next-generation phenotyping of electronic health records.” Journal of the American Medical Informatics Association 20(1): 117-121. A perspective piece, not a measured study. https://doi.org/10.1136/amiajnl-2012-001145
Agniel, D., Kohane, I. S., & Weber, G. M. (2018). “Biases in electronic health record data due to processes within the healthcare system: retrospective observational study.” BMJ 361: k1479. 669,452 patients at two Boston hospitals, 2005-2006; associations, not causes; the 118 of 174 figure counts only tests where either signal improved a baseline model. https://doi.org/10.1136/bmj.k1479
Goodhart, C. A. E. (1975). “Problems of monetary management: the U.K. experience.” Paper presented at the Reserve Bank of Australia conference, July 1975; published in Papers in Monetary Economics, Vol. I (RBA, 1976); reprinted in Monetary Theory and Practice (Macmillan, 1984). https://www.rba.gov.au/publications/rdp/1990/9013/conference-volumes.html
Strathern, M. (1997). “‘Improving ratings’: audit in the British University system.” European Review 5(3): 305-321. The popular phrasing of Goodhart’s law is hers, not Goodhart’s. https://doi.org/10.1002/(SICI)1234-981X(199707)5:3%3C305::AID-EURO184%3E3.0.CO;2-4
Chandra, T. D., & Toueg, S. (1996). “Unreliable failure detectors for reliable distributed systems.” Journal of the ACM 43(2): 225-267. The paper describes time-outs on periodic “is alive” messages; “heartbeat” is the common name for this pattern, not the paper’s term. https://doi.org/10.1145/226643.226647
Fischer, M. J., Lynch, N. A., & Paterson, M. S. (1985). “Impossibility of distributed consensus with one faulty process.” Journal of the ACM 32(2): 374-382. The crashed-or-slow line is an assumption of their model; the theorem is about consensus. https://doi.org/10.1145/3149.214121
Asimov, I. (1941). “Liar!” Astounding Science Fiction 27(3), May 1941; collected in I, Robot (1950). The 1941 magazine text already calls Calvin “the robopsychologist of the plant.” https://archive.org/details/Astounding_v27n03_1941-05
Cemri, M., Pan, M. Z., Yang, S., et al. (2025). “Why Do Multi-Agent LLM Systems Fail?” NeurIPS 2025, Datasets and Benchmarks Track. arXiv 2503.13657v3. Benchmark runs, not production; the taxonomy was built by human experts on about 150 traces and the full set was labeled by an LLM judge. https://arxiv.org/abs/2503.13657
Kerr, S. (1975). “On the folly of rewarding A, while hoping for B.” Academy of Management Journal 18(4): 769-783. The CRM example is my application; Kerr’s examples predate CRM software. The quoted wording is checked in Kerr’s 1995 reprint (Academy of Management Executive 9(1): 7-14), not the 1975 page. https://doi.org/10.2307/255378
ISO 17359:2018. Condition monitoring and diagnostics of machines: General guidelines. International Organization for Standardization (third edition, reconfirmed 2023). https://www.iso.org/standard/71194.html