Read enough postmortems and the pattern is unmistakable: almost no incident began as an incident. It began as something small, visible, and ignorable — and then it was ignored.
Most major incidents are the final stage of an escalation chain: a small error appears, repeats, compounds into degradation, degradation becomes failure, and failure becomes customer impact. Each link takes time — minutes to days — and each link is detectable. Proactive monitoring exists to interrupt the chain at its earliest, cheapest link, where the response is a routine investigation instead of an emergency. The economics are stark: the same root cause costs an engineer-hour at link one and a postmortem, an apology and churn at link five.
The canonical path from nothing to postmortem:
The chain's defining property is that time between links is opportunity: hours to days in which the problem is real, observable, and dramatically cheaper than it is about to become.
Attach rough costs to each link and the argument makes itself:
The multiplier between link one and link five is routinely two orders of magnitude. No other reliability investment buys cost reduction at that ratio, because no other investment gets to act before the expensive part.
Interrupting early needs the early links to be visible and credible:
Tooling makes the chain visible; a small practice makes the visibility compound. After every incident, ask the chain question: which link did we catch this at, and what signal would have caught it one link earlier? The answer is almost always specific — a fingerprint that existed on day one, an endpoint that deserved its own check, a trend that deserved a warning — and it becomes next sprint's monitoring change.
Teams that run this loop for a few quarters watch their incident distribution migrate leftward: fewer link-four surprises, more link-two investigations, and a growing file of would-have-been incidents that are now just tickets titled "investigated the warning, fixed the pool sizing". The major incident stops being a periodic certainty and becomes what it always technically was: a chain of small, catchable things — caught.
Most chains start with a change — correlating deploys with incidents is how to see the first link.
| The chain | Small error → repeated error → degradation → failure → customer impact |
|---|---|
| Time between links | Minutes to days — every link is an opportunity to act cheaply |
| Cost multiplier | Routinely ~100× between link-one investigation and link-five incident |
| Link 1–2 detectors | Error fingerprinting (new signature after deploy), rate deviation vs own baseline |
| Link 3 detectors | Latency trend against baseline, timeout onset, accelerating budget burn |
| Credibility rule | Early links get working-hours warnings, never 3am pages — or the system gets muted |
| The learning question | Which link did we catch it at — and what would have caught it one earlier? |
Through an escalation chain: a bug fails under a specific condition (small error), the condition recurs (repeated error), failures interact with the system — retries multiplying load, pools starving — producing degradation for everyone, degradation exhausts a resource into outright failure, and failure becomes customer impact. Each stage takes time, which is why early detection has room to work.
Because the early links emit signals most setups don't watch: a new error fingerprint, an error rate at multiples of its own baseline, latency trending upward while everything still "works". Uptime checks join the story only at outright failure — link four of five — and by then the cheap intervention window has closed.
Error collection with fingerprint grouping catches new failure signatures within minutes of a deploy introducing them. Baseline-deviation detection catches abnormal rates while absolute numbers are still small. Trend detection on latency and error-budget burn catches the compounding phase. Together they cover links one through three — the stages where a fix is an hour's work.
Because credibility is what keeps early detection alive. An early-link anomaly is a working-hours investigation, not an emergency; paging at 3am for it teaches the team to mute the system, after which the chain escalates unobserved. Severity honesty — only genuine emergencies interrupt — is a functional requirement of proactive monitoring, not politeness.
With one standing postmortem question: which link of the escalation chain did we detect this at, and what specific signal would have caught it one link earlier? The answer — a missing error collector, an unchecked endpoint, an un-alerted trend — becomes a concrete monitoring improvement, and repeated application migrates the team's incident distribution toward early, cheap catches.
No — some failures skip the chain entirely and arrive at link four with no prodrome: sudden infrastructure loss, instantly-fatal changes. Honest proactive practice targets the majority that do escalate gradually, while keeping fast reactive detection for the rest. The goal is shifting the distribution, not achieving a myth of zero incidents.
Merik is built as a chain-interruption system. Link one: merik.js reports new error fingerprints minutes after a deploy ships them, and GitHub/Vercel webhooks put the deploy right beside the spike. Link two: every rate is judged against that monitor's own 14-day baseline, with floors and confirmation so small numbers stay honest. Link three: trend detection on latency and error-budget burn — the compounding phase — feeds an early warning carrying risk, confidence and every contributing signal.
The credibility rules are structural: one warning per asset, one alert per incident, and only budget-burn emergencies may interrupt outside working hours — early links arrive as calm briefings, which is why they stay unmuted. And the learning loop is recorded for you: warnings link to the incidents they predicted, incidents carry the timeline of what preceded them, so "what would have caught this earlier?" has an answer in the data. The chain still starts sometimes; it just rarely gets to finish.
The failures nobody notices are the ones that compound. What silence costs, and how to end it.
Most incident time is spent not knowing. Six changes that cut detection from hours to minutes.
Deploys on the timeline, reading correlation honestly, and the rollback decision.