Incidents

How to reduce MTTD — mean time to detect — for application issues

Teams obsess over how fast they fix. The clock that usually dominates an incident is the one nobody watches: how long the problem ran before anyone knew it existed.

An incident timeline dominated by detection lag

MTTD — mean time to detect — is the average gap between a problem beginning in production and the team knowing about it. It is reduced by closing detection blind spots (frontend errors, per-endpoint failures, silent jobs), shortening check intervals with confirmation, alerting on deviation from baselines rather than hard failure, correlating signals so alerts stay credible, and routing findings to a named owner. For teams that rely on user reports or log archaeology, MTTD is measured in hours to days — usually the largest single component of incident duration, and the cheapest to cut.

Key takeaways
  • MTTD is the silent half of incident cost: damage accrues identically whether or not you know — knowing is what lets it stop.
  • The biggest MTTD driver is not slow tooling but blind spots: failure classes with no automatic detection at all.
  • Baseline-deviation alerting cuts MTTD below zero in effect — detecting deterioration before the failure it precedes.
  • Alert credibility is an MTTD input: a muted channel has infinite detection time, however fast the detector.
  • Detection ends when a responsible human knows — routing and ownership lag are part of MTTD, not after it.

Why MTTD dominates incident cost

Decompose incident duration: time to detect, then time to acknowledge, diagnose, fix, verify. Postmortems lavish attention on the later phases — the diagnosis that took an hour, the fix that needed a rollback. But for any team without systematic detection, the first phase quietly dwarfs them: a partial failure that runs from Friday evening to Monday's first complaint has an MTTD of sixty hours attached to a two-hour fix.

And detection lag is pure loss. During diagnosis and repair, at least the damage is being worked on; during the not-knowing, failed conversions, lost data and eroding trust accumulate at full rate with zero countervailing effort. Cutting an hour of MTTD is worth exactly as much as cutting an hour of repair — and it is almost always cheaper, because detection improves by configuration while repair improves by engineering.

What actually drives detection lag

Four causes, in descending order of typical impact:

  1. Blind spots. Whole failure classes with no automatic detector: frontend errors nobody collects, endpoints nobody checks individually, background jobs with no liveness signal, degradation below hard-failure thresholds. Here MTTD equals "whenever a human stumbles on it" — the ten-issue checklist is effectively a blind-spot census.
  2. Passive detection. The signal exists — in logs, in a dashboard — but reaches no one until queried. Recorded is not detected; a chart nobody is looking at has the same MTTD as no chart.
  3. Threshold lag. Detection waits for outright failure while the problem spends hours as measurable deterioration first. Hard-threshold alerting concedes all of that lead time by design.
  4. Credibility and routing decay. The alert fired — into a channel muted after months of noise, or addressed to everyone and therefore no one. Detection is not complete until a responsible human knows; sociology is part of the pipeline.

The six changes that cut MTTD

  1. Close the blind spots. Browser error collection, per-endpoint checks on money paths, liveness signals for background work, certificate monitoring. Each converts a detect-by-accident class into a detect-in-minutes class — the largest single MTTD improvements available, and mostly hours of setup.
  2. Make detection push, not pull. Every signal must evaluate itself continuously and notify on conclusion. Dashboards are for investigation, not detection.
  3. Check at the right frequency, with confirmation. Five-minute intervals with two-failure confirmation bound hard-failure MTTD at ~10 minutes; one-minute intervals buy ~2 minutes where SLAs demand it. Confirmation is what makes frequency affordable — without it, tighter intervals just page faster on blips.
  4. Alert on deviation, not just failure. Baseline-based detection moves the clock before the failure: latency at 3× normal or errors at 8× baseline is detection of the incident's prologue. This is where MTTD stops shrinking toward zero and effectively goes negative.
  5. Correlate before notifying. One cause, one alert, evidence attached. Credibility is the multiplier on everything above: a fast detector feeding a muted channel detects nothing.
  6. Route to a named owner. Every monitored asset has an owner; findings arrive assigned, severity decides urgency. The gap between "alert fired somewhere" and "the right person knows" is MTTD too — often the most embarrassing slice of it.

Measuring it honestly

MTTD only improves if the start of the clock is honest: when the problem began, not when the alert fired. Reconstruct onset from telemetry — the first anomalous check, the error spike's leading edge — and log detection time per incident alongside how it was detected (check, warning, user report). Three practices keep the number meaningful:

Detection time is only as good as the team's willingness to read the alert — alert fatigue is the failure mode that undoes everything above.

Quick reference

DefinitionMean gap between a problem beginning in production and the team knowing about it
Typical unmonitored MTTDHours to days — until a user report or accidental discovery
Largest driverBlind spots: failure classes with no automatic detector at all
Bounding formulaCheck interval × confirmation count — e.g. 5 min × 2 ≈ 10-minute worst case for hard failures
Going below zeroBaseline-deviation alerting detects the deterioration phase before failure
Detection endpointA responsible human knows — routing and credibility lag count as MTTD
Best single KPIShare of incidents first detected by your own systems vs by users

Frequently asked questions

What is MTTD?

MTTD — mean time to detect — is the average time between a problem beginning in production and the team becoming aware of it. It sits before acknowledgement, diagnosis and repair in the incident timeline, and for teams without systematic monitoring it is typically the longest phase, measured in hours or days.

How is MTTD different from MTTR?

MTTR (mean time to resolve/repair) usually measures from detection or acknowledgement to resolution — the visible, worked part of an incident. MTTD measures the invisible part before it: how long the problem ran with nobody knowing. Damage accrues through both, but MTTD's damage accumulates with zero countervailing effort, which is why cutting it is the cheapest reliability win available.

What is a good MTTD?

For hard failures, minutes: outside-in checks at a five-minute interval with two-failure confirmation bound detection at roughly ten minutes, and one-minute checks at about two. For degradation-class problems, good means detecting during the deterioration phase — before outright failure — which requires baseline-deviation alerting rather than hard thresholds.

How do I reduce MTTD fastest?

Close blind spots first: add browser error collection, individual checks on revenue-path endpoints, and liveness signals for background jobs — each converts a detect-by-accident failure class into detect-in-minutes. Then make every signal push-based, alert on deviation from baselines, correlate signals so alerts stay credible, and route findings to named owners. Blind-spot closure alone typically cuts average MTTD by more than any tooling upgrade.

Does alert fatigue affect MTTD?

Directly. A noisy channel gets muted, and a muted channel gives every subsequent incident effectively infinite detection time regardless of how fast the underlying detector fired. Correlation (one cause, one alert), confirmation before alerting, and severity-based routing are MTTD investments as much as courtesy.

How should teams measure MTTD?

Per incident, reconstruct the true onset from telemetry — the first anomalous check or the error spike's leading edge — and measure to the moment a responsible human knew. Track the average, but also the detection-source ratio: what share of incidents your own systems caught first versus users. That ratio is the clearest single indicator of monitoring maturity.

How Merik handles it

Merik attacks every term in the MTTD equation. Blind spots: browser errors via merik.js, per-endpoint checks with assertions, SSL expiry, vendor status — the classes teams usually discover by accident, detected automatically. Frequency and confirmation: checks every few minutes, two failures before an incident, so hard-failure detection is bounded in minutes without blip noise. Deviation: 14-day baselines per monitor turn deterioration into early warnings — detection before the failure phase begins.

Credibility and routing are enforced by design: correlated signals become one warning or one incident, alerts send once, severity gates what may interrupt outside working hours, and every asset has an owner who gets the assignment automatically. The scoreboard is built in too — incidents record how they were detected, and warnings link to the incidents they predicted, so the "caught by us vs caught by users" ratio is a number you watch improve, not a feeling.

Create your workspace →

Or talk to us about your team →