One missed webhook is not a broken integration

We pushed a large change to a client site and watched for the build. Nothing fired. After five minutes the newest production deployment was still three days old, so we deployed from the command line instead and moved on.

Two more pushes that afternoon, same pattern: push, then immediately deploy manually. By the third we had written it down as a fact. The git integration on this project is broken.

It wasn't.

What the evidence actually showed

When we finally investigated properly, the picture fell apart. Only the first push had genuinely been missed. The second and third had never been tested at all, because we ran the manual deploy in the same breath as the push. The manual deploy completed first, so of course no separate build appeared.

One dropped delivery, and two experiments we contaminated ourselves. From that we had concluded the integration was dead.

The test that settled it took two minutes: an empty commit, pushed on its own, with nothing else running. The build fired in eighteen seconds.

git commit --allow-empty -m "probe deploy trigger"
git push origin main

The signal we should have checked first

Deployment lists are a bad diagnostic. They lag, they paginate, and they show you the result of a process rather than whether the process began.

The authoritative signal is the commit status your host posts back to the repository. If it exists, the host saw your push. If it doesn't, the delivery never arrived. That distinction is the whole diagnosis, and it is one command:

gh api repos/OWNER/REPO/commits/SHA/status \
  --jq '.state + " " + (.statuses[0].description // "none")'

Run it across the last several commits and the pattern is immediate. In our case every commit had a status except one, which is the signature of a single dropped delivery rather than a broken connection. A broken integration produces a run of missing statuses, not a gap of one.

Why we got it wrong

Three failures stacked, and none of them were technical.

• We generalised from a single observation. One missed delivery is a transient; a pattern needs more than one data point, and we never gathered a second.
• We destroyed our own evidence. Running the manual deploy immediately after each push meant the two later pushes could never have demonstrated anything either way.
• We wrote the conclusion down. Once "the integration is broken" was stated as fact, it stopped being a hypothesis anyone would test.

The cost was small here, an hour and some unnecessary manual deploys. The same reasoning applied to a client incident produces a rebuilt integration nobody needed, or a vendor blamed for a problem they didn't have.

The rule we adopted

Before declaring infrastructure broken, run one clean experiment that isolates the thing you are testing. No parallel workarounds, no other commands in flight, nothing that could produce the outcome by another route.

If that feels like overkill for a two-minute check, notice that the alternative was three hours of working around a system that was functioning correctly the entire time.

A workaround that succeeds is the most expensive kind, because it removes the pressure to find out what was actually wrong.