A useful test of any monitoring setup: walk this list and ask, for each item, "would a machine tell us, or would a human have to notice?" Every "human" answer is a gap.
A production monitoring setup should automatically detect: full outages, API endpoint failures, error-rate spikes, JavaScript errors in browsers, latency degradation, expiring certificates, failed deployments' side effects, dead background jobs, third-party dependency outages, and slow resource exhaustion. Each has a distinct detection method, and none should depend on a person watching a dashboard. This article is the checklist form: what each issue looks like, why it evades casual observation, and the mechanism that catches it automatically.
1. The full outage. The site or app stops answering. Detection: outside-in availability checks every 1–5 minutes with consecutive-failure confirmation. The baseline capability — and still worth stating, because the check must run from outside: an internal probe happily reports a healthy process behind a dead load balancer.
2. The failed API endpoint. One route 500s or times out while the rest of the surface works; aggregate dashboards stay green. Detection: per-endpoint checks on every route that maps to money or a core journey, each with status and content assertions. The aggregate is where this one hides.
3. The expired certificate. The most preventable outage in the industry, still a weekly event somewhere, because auto-renewal fails quietly and nobody owns the calendar. Detection: daily TLS checks warning at 14+ days — converting a future outage into a routine chore.
4. The error-rate spike. Requests failing at several times the normal rate — after a deploy, under unusual input, from a struggling dependency. Detection: error-rate tracking per endpoint against its own baseline, with a floor so one blip is not a pattern. Fixed "alert at N errors" thresholds fail both busy and quiet services; deviation from measured normal is fair to both.
5. The latency climb. p95 rising steadily — growing table, filling pool, degrading dependency. No single moment is dramatic, which is why humans miss it live and find it retroactively in the postmortem. Detection: latency percentiles per endpoint against baseline, plus trend detection across hours: five consecutive rising hours is a signal in itself. This is the canonical proactive catch — hours of lead time, routinely.
6. Frontend errors in real browsers. The deploy that ships a JavaScript exception; the page that serves perfectly and breaks on arrival. Server-side signals: all green. Detection: in-browser error collection, grouped by fingerprint, alerting when a site's hourly rate is many times its own median. Without this channel, detection is "a user eventually emails".
7. The post-deploy regression. Not a failed deploy — a succeeded deploy that broke behaviour: latency up 3× on one route, errors trickling where there were none. Detection: deployment events recorded into the same timeline as anomalies, so any degradation is automatically read against "what shipped in the last hour". The correlation does not prove blame; it hands the responder the right first question.
8. The dead background job. Emails, exports, webhook processing, scheduled reports — producers keep succeeding, the consumer died Tuesday. Nothing in the request path errors. Detection: liveness monitoring on the work itself — queue depth, age of last completed run — exposed as a checkable endpoint. The classic silent failure, and the one most often discovered by a customer asking where their report went.
9. The third-party outage. Payments, email, hosting, CDN — their incident, your symptoms. Detection: consuming vendor status feeds and mapping which of your services depend on which provider, so your monitoring can say "checkout failing because the payment provider is down" — and can say it once, instead of paging per affected service. Debugging your own code during someone else's outage is a rite of passage nobody needs twice.
10. Slow resource exhaustion. Connections, memory, disk, quota — climbing toward a ceiling over days. At the ceiling it becomes issues 1–5 simultaneously. Detection: trend analysis on whatever resource signals are visible — and where direct internals are not observable, the external prodrome (latency drift and timeout onset) is the reliable proxy. The lead time here is the longest on the list, and so is the payoff for catching it.
A subtle failure mode of checklist-driven monitoring: implement all ten detectors independently and a single incident lights up half of them — the dependency outage (9) causes API failures (2), error spikes (4), frontend errors (6) and a latency climb (5), producing five streams of notifications about one fact.
Teams rationally respond to that noise by muting things, and a muted channel catches nothing. So the checklist has an eleventh, structural requirement: correlation. Signals that share a cause should merge into one incident carrying all the evidence; suppression should apply when a hard dependency is publicly down; and severity should decide loudness, so a small team's on-call is interrupted only by things worth interrupting for. Detection coverage gets you to "the machine noticed"; correlation gets you to "and it told us exactly once, with the story assembled" — which is the actual goal. The economics of that gap are covered in reducing MTTD.
Two of the ten have their own guides: certificate expiry and deploy-caused regressions.
| Hard failures (1–3) | Full outage, single failed API endpoint, expired TLS certificate |
|---|---|
| Degradations (4–6) | Error-rate spike vs baseline, latency climb vs baseline, browser error spike |
| Silent stoppages (7–10) | Post-deploy regression, dead background job, third-party outage, slow resource exhaustion |
| Longest lead time | Certificates (weeks) and resource exhaustion (hours–days) |
| Shortest lead time | Full outage — minutes, via confirmed outside-in checks |
| Structural requirement | Correlation: one underlying cause must produce one incident, not five alert streams |
At minimum: full outages, individual API endpoint failures, error-rate spikes, JavaScript errors in real browsers, latency degradation against baseline, expiring TLS certificates, regressions following deployments, dead background jobs, third-party dependency outages, and gradual resource exhaustion. Each has a distinct automatic detection method; none should rely on a person watching dashboards.
Because roughly half the common production issues are invisible to an availability check: browser-side errors, single failing endpoints inside a healthy aggregate, dead background consumers, latency degradation and slow resource exhaustion all leave "is it up?" green. Coverage of those channels — per-endpoint checks, frontend error collection, job liveness, baselines — is what closes the gap.
By monitoring the work rather than the request path: expose queue depth and the age of the last successful run as a small status endpoint, and check it like any other endpoint. Producers succeeding while the consumer is dead generates no errors anywhere, so without an explicit liveness signal this failure is typically discovered by a customer.
Record every deployment as an event on the same timeline as checks and anomalies, via a webhook from your CI or hosting platform. When latency or errors deviate shortly after a deploy, the correlation is presented with the incident — as context for the responder's first question, not as automatic blame.
Correlate before notifying: merge signals that share a cause into one incident carrying all the evidence, suppress alerts for services whose hard dependency is publicly down, and let severity decide what interrupts versus what waits for working hours. Detection without correlation produces alert fatigue, and muted channels catch nothing.
Certificate expiry (known weeks ahead) and slow resource exhaustion (visible as trends over hours to days) offer the longest lead times. Latency drift against baseline typically gives hours. The value of proactive monitoring is concentrated in these long-lead signals, because they convert would-be outages into scheduled work.
Merik's Digital Operations module covers this list as shipped behaviour: confirmed outside-in checks (1), per-endpoint monitoring with status and content assertions (2), daily SSL expiry warnings (3), error-rate and latency deviation against 14-day baselines with trend detection (4, 5, 10's external prodrome), the merik.js browser SDK (6), GitHub/Vercel deploy correlation on the incident timeline (7), and live vendor status feeds with hard-dependency suppression (9).
The eleventh requirement is the architecture, not a feature: correlated signals produce one early warning per asset with the evidence attached, incidents open once, alert once, and arrive assigned to the asset's owner. Background-job liveness (8) is yours to expose as an endpoint — and once exposed, Merik monitors it like anything else. Walk the checklist against your own setup; where the answer is "a human would have to notice", the full checklist article shows the fix.
APIs degrade before they die — latency drift, timeouts, error trickles. Monitoring that listens.
The server said 200. The page broke anyway. Seeing the failures uptime checks can't.
200-day certificates now, 47 by 2029. Renewal fails silently; monitoring catches it.