The Check That Cried Wolf: When an Accurate Alert Is Worse Than No Alert
A continuous-integration check we rely on failed every pull request on three of our published projects for about a day. None of those pull requests had anything wrong with them.
The engineer kept merging past it. That was the correct call each time — the check was wrong and the work was clean. It is also precisely how a control stops working.
The failure was in a different repository
The gate reads its logic from the base commit rather than from the pull request. That is deliberate: otherwise a pull request could supply the code that judges it, which is not a gate at all.
The side effect is not obvious. The script that actually runs is the copy already on the default branch, so a defect there breaks every pull request in that repository, and nothing done to the pull request helps. Rebasing does not help. Fixing the commit message does not help. The check is reporting on the branch and failing because of somewhere else.
The underlying bug was small: each commit message was compared against a copy that had gained a trailing newline, so every commit looked modified and the run aborted. The fix was a few lines. Finding it took a day, because everyone was looking at the pull requests.
The general rule worth extracting: the thing that judges must exist before the thing being judged. When a control reads its logic from a trusted location, that location is updated first. We now land the scripts on a repository before the workflow that calls them, and we merge the canonical copy before any copy that vendors it.
An accurate finding nobody could clear
Separately, one of our governance modules reported an error on every environment, permanently, by design.
A role had been granted blanket administrative rights, and the module correctly refused to certify that such a role was constrained by policy. Because the role held every permission implicitly — including permissions that did not exist yet — there was nothing to enumerate and nothing to sign off. The finding could not be acknowledged. It simply stayed on the status report.
The reasoning at the time was defensible as far as it went. The alternative under discussion was deleting the policy that produced the finding, and that would have quietly removed a real network boundary along with the red row. Keeping the noise was better than removing the control.
But it was never an acceptable end state, and it sat that way longer than it should have. A report that always shows one error teaches its reader to skip the report. The finding was accurate, and that is what made it corrosive: nobody could act on it, so everybody learned to look past it. The next error would have arrived into that habit.
The question nobody had asked was why the role held blanket rights at all. Once asked, it took a single change — replace the blanket grant with an explicit list of permissions. The role became something the policy could constrain, the finding cleared honestly, and the report went back to meaning something.
A verification that could not fail
The same week we needed to confirm that a hundred recorded decisions had been carried into a new knowledge base. A script compared them and reported ninety-nine already present.
Before believing it, we ran it against one decision we already knew was missing. It scored that one 0.88 — comfortably a pass.
The script matched vocabulary, not meaning. Words from the missing decision appeared throughout the corpus in unrelated contexts, so it declared the decision present. We discarded the result and read all hundred by hand. Twenty-seven were genuinely absent.
A verification that cannot produce a failure is not a verification. Every check should be shown to fail on a known-bad input before it is trusted on unknown ones. It costs one test, and it is the difference between a measurement and a reassurance.
Why this is not alert fatigue
These look like three unrelated bugs. They are one failure mode in different clothes, and it is worth separating from the more familiar problem it resembles.
Alert fatigue is usually described as a volume problem — too many notifications, attention exhausted. None of these were high volume. Each was a single, persistent, correct-looking signal. What made them harmful was that no action available to the reader would clear them:
- The failing gate could not be satisfied by changing the pull request, because its cause was elsewhere.
- The governance error could not be acknowledged, because the code path returned before acknowledgement was read.
- The verification could not report a problem, because its method was incapable of detecting one.
In every case the control remained present. It appeared in dashboards and status checks. An auditor reviewing the configuration would have found it enabled and correctly configured. What had quietly ended was its effect on anyone's behaviour.
That gap — between a control that is configured and a control that is working — is the one we look for first when assessing an environment.
Three practices
Treat a persistently red check as an incident. If a control has been failing for days while work ships anyway, the control is already not working, whether or not its findings are correct. Fix it or remove it. Leaving it in the middle is the worst of the three options, because it looks like coverage.
Require that every finding has an action that clears it. A fix, an acknowledgement, or an exemption with a named owner and an expiry date. A finding with no path to resolution is a design defect in the control, even when the finding itself is true. If your tooling can emit a permanent unclearable state, that is a bug in the tooling.
Prove each check can fail. Give it a known-bad input and confirm it reports. A test suite that only demonstrates passing tells you the check runs, not that it works. This applies to the checks themselves and to the tests that cover them.
Why we are writing this down
We build governance and compliance tooling and we run it on our own systems before anyone else's. That means we find these problems in our own environment first, and we would rather describe them plainly than present a tidier account of how the work goes.
The uncomfortable part is that two of the three examples were defects in tooling built to catch defects. Controls are software. They fail the way software fails, and they fail quietly, because a broken control is indistinguishable from a working one until something tests it.
Configuration asserting a control it does not enforce is the shape underneath all of this: a ticked checkbox, a present policy, a check that runs — and nothing behind any of them. It is the first thing we look for in someone else's estate, largely because of how often we have found it in our own.