Field report · details redacted for publication

It worked for months. Then we checked, and it didn't.

This describes a real, running multi-agent system, not a hypothetical one, and a real failure it had. Some specifics below are deliberately left out. Not because they're complicated. Because they're the part worth paying for.

The mechanism is not the moat. Knowing which mechanism to trust, and when to stop, is.

FIELD TESTED

What actually happened

A review process that quietly stopped doing its job

One process had been verifying finished work reliably for a long time, across a lot of runs. It was never rebuilt, never rewritten, just trusted, the way a working thing earns trust: by continuing to work. Then a second, structurally different kind of build came along, one that looked close enough to the first that reusing the same review process seemed like the obvious move.

It wasn't. the specific check that exposed it confirmed the whole approach that process depended on had no path to the new case at all. Not a partial gap. Not an edge case. A structural dead end, the kind you only find by actually running the thing against the real target, not by reading the code and reasoning about it.

Close enough looked true right up until someone actually checked.

The choice

Building two things, and refusing to build a third

The cheap fix is always available: stretch what already works to also cover the new case, patch it until it passes, call it done. That fix got rejected on purpose, once direct testing confirmed the two cases needed genuinely different tools underneath, not one tool wearing a different label.

A second, purpose-built process got made for the case that actually needed one. A third case, superficially similar to both, got evaluated the same rigorous way, and got nothing built for it, because the job it needed doing was already covered by something already in place. Nobody added a process just to make the roster look symmetric.

NOTE: the specific technical reason the third case needed nothing new is withheld here. The reasoning pattern that led to that conclusion is not: check first, build only where checking says you must, and be willing to have that answer be "nothing."

The harder discipline isn't building the second thing. It's refusing to build the third.

Why the fix stayed small

The rest of the system never noticed

Rebuilding one piece of a system this size is normally where things get dangerous: one change ripples outward into every place that quietly assumed the old piece would keep behaving a certain way. That didn't happen here.

The piece that changed had always been sealed behind a boundary the rest of the system was never allowed to reach past. Nothing outside it had to be touched, updated, or even reviewed, because nothing outside it had ever been permitted to depend on how that piece worked internally in the first place. The boundary wasn't added for this fix. The fix was only cheap because the boundary already existed.

A boundary you actually enforce is what makes a rebuild small instead of contagious.

Knowing when to stop trusting the loop

The same finding twice is a different problem than once

Every review-and-retry loop eventually has to answer an implicit question: is this converging, or is it stuck? The system doesn't guess at the answer. A finding that resolves the first time it's raised is ordinary work. The identical finding showing up again, after it was supposedly addressed, gets treated as evidence about something upstream, not as a reason to try the same pass one more time.

Past that point, the system stops retrying on its own and puts a person in the loop instead. On purpose. Every time. No exception carved out because stopping felt inconvenient in the moment.

Knowing the difference between still-converging and stuck is worth more than any single fix inside the loop.

This is what operating experience actually looks like on paper

A tutorial tells you every step because the steps are the whole value. A field report tells you what broke, what the fix cost, and what it revealed, because the value was never the steps. It was running the thing long enough, and carefully enough, to find out where it actually breaks before a customer does.

That's not reproducible by reading a page. It's reproducible by doing the same amount of real work this took, which is the entire point of writing it down this way instead of as a how-to.

If a write-up tells you everything you'd need to rebuild the system exactly, ask what that says about how much real operating experience actually went into writing it.

For the record

Is this a real system, or a design exercise?

Real, running, in active use, with a version history and live test runs behind every claim on this page.

Could someone rebuild this from what's shown here?

The rough shape, maybe. The judgment calls that make the shape hold up under a real, unexpected failure, the part that actually took the work, no.

Is this specific to one technology stack?

No. Everything here is about how independent pieces get coordinated and trusted, not about any one framework or language underneath them.

This report is one artifact of a larger practice.