The monitoring was set up carefully. Three months later the channel is muted and the outage is discovered by a customer. Nothing broke in between except the ratio of alerts to problems.
Alert fatigue is what happens when the number of alerts exceeds the number of problems by enough that reading them stops being worth it — and in a small team that point arrives fast, because the same three people receive every alert. The causes are structural: one alert per signal instead of per problem, static thresholds that fire on normal variation, no deduplication so one outage produces twenty messages, flapping checks that open and close every few minutes, and no notion of quiet hours. The fix is equally structural: one alert per problem, carrying its evidence, sent to a named owner, once, with recovery closing it automatically. Then measure two numbers — alerts per week, and the fraction that led to action — and keep the second above half.
| 1. One alert per problem | Correlate signals from the same asset into one incident with an evidence list. Latency + errors + failed check = one message that lists all three. |
|---|---|
| 2. Judge against normal | Baselines per asset — this endpoint's own p95, its own error rate — so the alert means "abnormal for this", not "above a number someone typed". |
| 3. Confirm before alerting | Two or three consecutive failures before a check counts as down. Kills flapping and single blips. |
| 4. One owner, once | Auto-assign to the asset's owner and alert once by email or Slack. Escalation, if needed, is a separate rule, not a repeat. |
| 5. Quiet hours and severity | Non-critical assets wait for morning. Critical ones do not. The person receiving the alert should be able to tell which from the first line. |
| 6. Auto-close on recovery | An incident that resolves itself closes itself, with the duration recorded. No manual tidy-up, no stale alerts. |
An alert that says "latency high" makes the reader do the investigation. An alert that says "p95 latency 2.4 s vs 14-day baseline 410 ms; error rate 3.1% vs 0.2%; a deploy landed 9 minutes before; no provider incidents reported" has done the first fifteen minutes of the investigation already — and, crucially, tells the reader whether to get out of bed. Fewer alerts with evidence beat more alerts without it, every time. Add a confidence score separate from the severity: "high risk, low confidence" is honest and useful; a single number that blends the two is neither. Where learned baselines and correlation help.
Alerts per week, per person. Action rate — the fraction of alerts that resulted in someone doing something (a fix, a rollback, a ticket, a decision to accept). Review both monthly. Action rate under 50% means half the alerts should not exist; find the noisiest source and apply the six rules to it. A small team with well-tuned alerting typically sees a handful of alerts a week, nearly all actionable — which is a team that reads its alerts. And a team that reads its alerts detects in minutes.
| Definition | Alerts outnumber real problems until reading them is not worth it |
|---|---|
| Threshold | ≈ 3:1 alerts to problems people stop reading; ≈ 10:1 they mute |
| Top cause | One alert per signal instead of per problem |
| Six rules | One per problem · baselines · confirm first · one owner once · quiet hours + severity · auto-close |
| Evidence | Every alert carries the signals that produced it and a separate confidence score |
| Metrics | Alerts/week per person; action rate ≥ 50% |
Alert fatigue is the state in which monitoring alerts arrive so much more often than real problems that the people receiving them stop reading, and eventually mute them. It is caused by the structure of the alerting — one alert per signal, static thresholds, no deduplication, flapping checks, no quiet hours — rather than by the team's diligence.
Correlate all signals from one asset into a single incident with an evidence list; judge each signal against that asset's own baseline rather than a fixed threshold; require two or three consecutive failures before alerting; send one alert to one named owner rather than re-paging; apply quiet hours and severity so non-critical assets wait for morning; and close incidents automatically when the checks recover.
There is no fixed number, but the ratio matters: when alerts outnumber real problems by more than about three to one, people stop reading them. Track alerts per week per person and the fraction that led to action; if fewer than half were actionable, the alerting needs to be fixed at the source.
Because they measure distance from a number rather than distance from normal. A threshold set at 800 ms fires every busy morning when 800 ms is normal load, and stays silent at 2am when latency triples from 150 ms to 450 ms. Baselines per asset — its own latency percentiles and error rate over a trailing window — alert on what is abnormal for that asset.
Not by default. Repeated paging is the fastest route to a muted channel. Alert once to a named owner with the evidence; if escalation is genuinely required for a critical asset, make it a separate, explicit rule with a longer interval rather than a repeat of the same message.
The asset and what is abnormal about it, the signals that produced the alert with their current and baseline values, correlated context such as a recent deploy or a provider incident, a severity, a separate confidence score, and the owner. An alert with that content has done the first fifteen minutes of investigation before anyone opens it.
Merik's Digital Operations module is built around the six rules. Each asset's latency and error-rate baselines are measured over 14 days, so warnings mean "abnormal for this asset". Correlated symptoms produce at most one early warning per asset, carrying a risk score, a separate confidence score and the evidence list that produced them — including a deploy that landed just before, from GitHub or Vercel webhooks, and any provider incident from vendor status feeds. Failures are confirmed before an incident opens.
Incidents are auto-assigned to the asset's owner and alerted once by email or Slack, with quiet-hours rules; warnings that recover close themselves, and warnings that came true are linked to the incident they predicted so you can see the action rate for yourself. See the Digital Operations module.
Most incident time is spent not knowing. Six changes that cut detection from hours to minutes.
The first ten minutes, what not to do, honest communication, and shrinking the blast radius.
Detecting problems before users report them — baselines, anomalies and early warnings, explained.