Green Is Not Evidence: The Deploy Failures That Never Fail
A deployment pipeline that fails loudly is doing its job. The dangerous ones are the deployments that succeed while accomplishing nothing.
Over a single evening of maintenance on one platform, we found six separate defects. Not one raised an error. Every affected deployment finished green. Several had been running that way for weeks, one for over a month.
They turned out to be the same bug wearing six costumes.
The six
A DNS server restarting on every deploy. The internal zone file derived its serial number from the clock. Any deployment landing in a new hour rendered a byte-different file, so the configuration management tool reported a change and restarted the resolver. No DNS record had changed. Each restart was a brief window where internal name resolution could fail — during a deployment, which is precisely when other steps are resolving those names.
A database container replaced on every deploy. A build step ran with provenance attestation enabled, which stamps the image manifest per invocation. A fully cached build — every layer reporting zero seconds — still produced a new image ID. The converge step then did exactly what it was designed to do: adopt the changed image and replace the container. The result was a cold database start on every deployment, in a system with an explicit, documented guarantee that this would not happen.
A nightly security scan that had never run. The container vulnerability scanner resolved its runtime through a search path that did not contain it. It failed every night at 4am into a log nobody read. The fix already existed in a sibling script, complete with a comment explaining the exact failure. It had never been applied here.
Staging deployments converging production. A configuration variable had been renamed. The automation read the new name; the deployment pipelines still passed the old one. The unknown variable was silently ignored and the scope fell back to its default — which was "everything." Staging deployments had been converging the production stack, and production promotions the staging stack, for weeks. Both reported success.
A configuration flag discarded on import. A new setting was added, validated against its schema, and exported correctly. The content framework only persists properties declared in an entity's export list, and the property had not been added there. The value was written, validated, and thrown away. The status command reported the entity as permanently out of sync — which reads as a warning nobody acts on.
A fix applied to code that was not running. A security improvement was made to a plugin that turned out to be disabled in configuration. The change was correct, tested, reviewed, and irrelevant. The live code path was somewhere else entirely.
What they have in common
In every case the system reported success, and the report was narrowly true. The deployment did run. The configuration did import. The build did complete. The scanner did execute.
What none of them reported was whether the intended outcome occurred.
That is the difference between a process check and an outcome check, and most automation only does the first. An exit code tells you a command ran. It does not tell you the command did anything.
Loud failures are the lucky ones
A crash gets fixed. It interrupts someone, produces a stack trace, lands in an alert. The feedback loop closes in minutes.
A silent no-op has no feedback loop. It accumulates. The security scan had not run in an unknown number of weeks — the error log went back further than anyone had looked. The staging-converges-production bug was invisible precisely because the extra work was usually harmless: it changed nothing most of the time, so it left no trace.
These defects also survive review. Reviewers read the change, not the runtime. A patch applied to a disabled plugin looks identical in a diff to a patch applied to live code.
The practice that catches them
Make "changed" mean something. In configuration management, a step that reports a change on every run is not reporting anything. Several of these bugs hid behind steps structurally incapable of reporting "no change" — a shell command with no condition attached always looks like it did something. Once those were corrected, the genuine churn became visible immediately, because it was the only thing still moving.
Make generated artifacts functions of their inputs. The DNS serial derived from the clock; the container image derived from the current timestamp. Neither was a function of the thing it described. When a generated file changes only when its inputs change, "this file changed" becomes a signal rather than noise.
Verify by exercising behaviour, not by reading exit codes. This matters most and gets skipped most, because it is slower. For the discarded configuration flag, every available indicator was healthy: deployment green, schema valid, file correct on disk. The only check that distinguished working from broken was performing a real content edit and watching whether the system responded. It did not.
What it costs
Outcome verification is slower than reading a status code. That is the honest trade. It means someone — or something — doing the thing the system is supposed to do and confirming the effect, rather than confirming that a command exited zero.
The alternative is a system where the dashboard is green, the backups are configured, the scanner is scheduled, the secrets are rotated, and some unknown subset of that is not true. Most organisations do not learn which subset until an audit, an incident, or a restore.
None of these six caused an outage. Every one was found by looking past a green check. Every one would still be running otherwise.