Production is the referee: a claim about 'why' isn't settled by who's most senior
Somebody senior says, with total conviction, "the flakiness is coming from the retry logic, not the queue." Nobody at the table has data on this. Nobody argues either, because arguing with the most senior, most experienced, most obviously-right-more-often-than-you person in the room about a technical judgment call is a genuinely bad career move, and everyone at the table knows it without having to think about it. The meeting moves on. The claim gets treated as settled.
It isn't settled. It's just unopposed, which is a completely different thing wearing the same clothes.
Why nobody in the room actually argues
It's worth being honest about why the silence happens, because it isn't laziness or agreement — it's a rational read of the room's social cost structure. Amy Edmondson's 1999 Administrative Science Quarterly paper on psychological safety in work teams found that a team's willingness to voice a dissenting technical opinion depends heavily on whether members believe doing so is interpersonally risky — will this make me look like I don't trust a colleague, will it slow the meeting down, will it be held against me later. Junior engineers in particular tend to correctly perceive that the expected cost of publicly contradicting a senior colleague's confident causal claim is higher than the expected cost of quietly letting it stand and finding out later whether it was right. Nobody has to be intimidating for this calculus to hold. The mere existence of a status gradient in the room is enough.
This is exactly why a claim ledger has to do the disagreeing that the room won't. It doesn't need anyone to raise a hand and say "I'm not sure that's the queue." It just needs the claim logged, and the telemetry to eventually have an opinion neither the room's silence nor the room's psychological-safety problem can suppress. The correction arrives on its own schedule, after the meeting, from a source with no stake in anyone's standing.
The acronym that named this at Amazon and Microsoft
Ronny Kohavi spent years running large-scale online controlled experiments — the rigorous cousin of the A/B test — first at Amazon, then at Microsoft's Bing and Experimentation Platform teams, and coined a term for exactly this failure mode: HiPPO, the Highest Paid Person's Opinion. His documented, repeated finding, across thousands of tested feature ideas from genuinely smart, experienced people, is that a strikingly large share of confident predictions about what will improve a product turn out, when actually measured against real user behavior, to be wrong or simply undetectable. Not because those people were foolish — because confident intuition about a complex system, however senior the intuitor, is a hypothesis, and hypotheses have a base rate of being wrong that seniority does not repeal.
Kohavi's argument, laid out on his own research site under the HiPPO name, is not "ignore experts." It's narrower and sharper than that: let experts propose the hypothesis — that's exactly what expertise is good at — and then let the system, not the expert's rank, decide whether the hypothesis survived contact with what actually happened. The org chart gets a vote on what gets tested. It does not get a vote on the result.
The same discipline, aimed inward
Site reliability teams reinvented a version of this for their own use, independently, because production systems punish deference just as reliably as product experiments do. Google's own site-reliability engineering handbook, in its chapter on monitoring distributed systems, argues for building alerting and diagnosis around measured symptoms — the thing the system is actually doing — rather than guessed causes, precisely because guessed causes, confidently asserted at 3 a.m. by whoever's paged, have a well-earned reputation for sending the whole incident response down the wrong branch. The discipline isn't distrust of expertise. It's a refusal to let expertise skip the step where it gets checked.
What "resolved by telemetry" actually means
A claim like "the flakiness is the retry logic, not the queue" is not a fact. It's a causal hypothesis about which code, once changed, will move a measurable number — and the number in question already exists, sitting in whatever's been logging fault rates on both pieces of code the whole time. A prediction like this resolves by looking at exactly that: did the fault rate on the named function actually move, in the direction the claim predicted, once the corresponding change landed. Not "did the person sound sure." Not "did the room agree." Did the number the claim was actually about do what the claim said it would do.
Notice what this resolution can and can't tell you, because the honest boundary matters. If the fault rate on the retry function drops after the change and the queue's fault rate is unaffected, the claim resolves came_true — the senior engineer's hypothesis survived contact with production. If both numbers stay flat, or the queue's fault rate drops instead, the claim resolves did_not, regardless of how the conversation in the room went. And if the change never actually shipped in isolation — bundled with four other fixes in the same release, fault rates moving for reasons nobody can cleanly separate — the honest outcome is unresolvable, not a guess dressed up as a verdict. Telemetry is a referee, not an oracle; it can only rule on what it can actually see cleanly; the previous part of this arc exists precisely because that third outcome has to stay on the board.
This is, in a sense, the least glamorous conclusion in this entire arc, and it's worth stating without dressing it up: most causal claims made confidently in a standup are not actually testable in the clean, single-variable way a controlled experiment would demand. Production is messy, changes bundle, and the world keeps moving while you're trying to isolate one cause. Telemetry-as-referee doesn't promise a clean verdict on every claim, ever. It promises that when a clean verdict is available, no amount of seniority overrides it — and that when it isn't available, the system says so honestly instead of manufacturing false certainty on either side.
That honesty has a name already established two parts ago in this arc: unresolvable. A causal claim tangled up in a bundled release, or made about a function whose telemetry simply wasn't being watched closely enough at the time, doesn't become did_not by default just because nobody can prove it true. It joins the same third bucket every other unresolved claim in this arc joins, for the same reason — because guessing at a verdict production genuinely cannot supply is exactly the false precision the whole approach exists to refuse.
Seniority earns you the floor. It has never once, in any of these literatures, been shown to earn you the verdict. Only the system being described can do that.
Why this protects seniority, not just checks it
There's a version of this argument that lands as an attack on experienced engineers, and that version is wrong. The best senior engineers are, overwhelmingly, the ones whose intuitions telemetry keeps confirming — which is precisely why their confidence reads as trustworthy in the first place, and precisely why it's worth measuring instead of just assuming. A calibration score built this way doesn't erode a good senior engineer's standing. It's the only mechanism that can ever prove, with something other than tenure, that the standing was earned. The engineer whose gut has been quietly, verifiably right for three years gets to point at that instead of at a job title, which is a considerably stronger thing to be able to point at.
And it works in the other, more useful direction too: the loudest voice in the room and the correct voice in the room are, empirically, two different populations with partial overlap, and the only honest way to find out which population any given person belongs to is to let production keep score, quietly, over enough claims to mean something (in CodeNSM's own fleet telemetry, that population overlap is real but far from complete — confidence and correctness travel together often enough to be dangerous, and separately often enough to matter). Nobody has to be embarrassed in the meeting for that to happen. The number just has to exist.
There's a habit worth adopting even without any of this instrumented: the next time a confident causal claim lands unopposed in a standup, write it down as a hypothesis with a name attached, rather than let it settle into "what we now believe." Not to catch anyone out — to give production, later, a specific, falsifiable sentence to referee instead of a vague institutional memory that "we looked into it and it was the retry logic." The habit costs ten seconds. The alternative costs a repeat incident, a year later, on the exact function everyone thought had already been explained.