Reliability is not a tool you install; it is a loop you run: observe, understand, predict, warn, respond, learn. This guide assembles the whole loop — and links to the deep dives for every part.
Proactive application reliability is the practice of keeping software dependable by detecting deterioration before it becomes failure — combining outside-in monitoring, measured baselines, early warnings, error budgets, correlated incident response and post-incident learning into one continuous loop. It differs from traditional reactive operations in when work happens: reactive teams spend their effort during and after incidents; proactive teams spend a fraction of that effort earlier, when the same problems are still cheap trends. This cornerstone guide assembles the complete practice and links to the detailed articles on each component.
Every mature reliability practice, whatever tooling it runs on, is the same six-stage loop:
Reactive operations run stages 5–6 only. The proactive difference is stages 2–4 — and they are precisely the stages that require no heroics, just accumulated data and honest statistics.
Observation must start where users are: outside-in checks on every user-facing surface, per-endpoint API monitoring with status and content assertions, browser error collection for the layer servers cannot see, certificate and DNS checks for the calendar failures, and deployment events recorded into the same timeline. (The full inventory, with priorities, is the SaaS monitoring checklist; the minimal version for tiny teams is the startup edition.)
Stored observations become baselines: latency percentiles, error rates and volume per monitor over a trailing window. Baselines are the hinge of the whole practice — they convert "is 700ms slow?" from a debate into a lookup. Two weeks of five-minute checks yields a stable normal; from then on, abnormal is computable. The proactive monitoring guide covers the mechanics.
Detection is comparison: current behaviour against baseline (is p95 at 3× its own normal?), and trend against time (has latency risen five hours straight?). The art is separating signal from weather — ratio and absolute thresholds, confirmation before conclusions, floors under error rates — so that what surfaces deserves attention. Ten specific detections cover the practical catalogue.
Warning is the part most tooling gets wrong, in one of two directions: alert-per-signal (noise, then muting, then nothing) or a single opaque score (unexplainable, then ignored). The honest form is one warning per degrading asset, carrying:
And warnings must be accountable: self-resolving when behaviour recovers, linked to the incident when they come true, so the system's hit rate is measurable. A prediction system that never admits misses is a horoscope. Where AI helps and where it overclaims is its own discussion.
Proactive practice needs a quantitative definition of "how reliable is reliable enough", and the SRE tradition's answer — the error budget — remains the best available. Declare a target per service (99.9% monthly availability); the target implies allowed failure (0.1% ≈ 43 minutes); the budget is what remains of that allowance; health is budget remaining, a number explainable in one sentence and traceable to raw checks.
Burn rate — current spend speed as a multiple of the break-even rate — is the severity signal: sustained 14× burn over an hour is an emergency arithmetically, not by anyone's gut feel. Budgets also settle the classic tension between shipping and stability: a healthy budget licenses risk-taking, an exhausted one buys down risk instead. The full construction is in the application health guide.
Early detection buys minutes; response design decides whether they are spent or wasted. Four rules preserve the head start:
The loop closes when incidents feed back into detection. After each one: what preceded it in the telemetry? Was there a warning — and if not, what signal would have caught it? Which warnings this month came true, and which fizzled? This review is what tunes thresholds, adds missing monitors, and builds institutional memory — interrupting the escalation chain earlier each time.
Run for a few months, the loop compounds visibly: baselines sharpen, warnings grow more precise, repeat incident classes get monitors and stop repeating, and the on-call experience shifts from firefighting to reviewing briefings. The destination is not zero incidents — that target is dishonest — but a steadily larger fraction of problems met as trends rather than outages, with reactive capability intact for the failures that give no warning. Reliability, in the end, is the loop running.
The loop's output to a client is the monthly SLA report; its output to a user is the status page.
| The loop | Observe → establish normal → detect abnormal → warn early → respond → learn |
|---|---|
| Observation layers | Outside-in checks, per-endpoint APIs, browser errors, certificates/DNS, deploy events |
| The hinge | Baselines — measured normal per monitor, making "abnormal" computable |
| Honest warning form | One per asset: risk + confidence separately, evidence with magnitudes, correlated context, recommendation |
| Arithmetic backbone | Error budgets: health = budget remaining; burn rate = severity |
| Response rules | Correlate first, assign by ownership, attach context, severity decides loudness |
| The goal | Not zero incidents — a growing fraction of problems met as trends instead of outages |
It is the practice of keeping software dependable by detecting deterioration before it becomes failure: continuous outside-in observation of what users experience, baselines defining measured normal, deviation and trend detection, evidence-backed early warnings, correlated incident response routed to named owners, and post-incident learning that improves detection. The defining difference from reactive operations is when the work happens — before failure, while problems are still trends.
Monitoring is the observation stage — necessary but only one-sixth of the loop. A reliability practice adds baselines (what is normal), detection (what is abnormal), honest warning (what deserves attention, with what evidence), structured response (one owned incident per cause) and learning (what would have caught this earlier). Many teams have monitoring; far fewer run the loop.
An error budget is the failure a reliability target permits — a 99.9% monthly target allows about 43 minutes of downtime. Expressing health as budget remaining makes reliability arithmetic rather than argument: severity comes from burn rate, priorities come from budget state, and the ship-versus-stabilise tension resolves by looking at the number.
Yes — every stage has a low-maintenance form. Outside-in checks and certificate monitoring are minutes of setup; browser error collection is a script tag; baselines accumulate automatically from stored checks; and correlation plus ownership are properties of good tooling rather than headcount. Coverage of the loop matters more than sophistication at any stage.
No, and honest practice never claims it. Some failures arrive with no warning. The aim is to move the substantial fraction of incidents that are preceded by measurable deterioration — latency drift, error creep, resource exhaustion, expiring certificates — from the outage column to the caught-early column, while keeping fast reactive detection for the rest.
Start the observation layer this week: outside-in checks on user-facing surfaces, per-endpoint checks on money paths, SSL monitoring, browser error collection, deploy events. Let two weeks of history build baselines, then enable deviation-based warnings. Add error-budget targets per service, assign owners to every monitored asset, and begin monthly incident review. Each step is small; the compounding is in running the whole loop.
Merik's Digital Operations module is this loop, shipped as a product. Observation: outside-in checks with confirmation, per-endpoint assertions, daily SSL monitoring, the merik.js browser SDK, and deploy events from GitHub/Vercel webhooks. Baselines: p50/p95/p99 and error rates per monitor over 14 days, recomputed hourly. Detection and warning: at most one early warning per asset, carrying risk and confidence separately with the full evidence list, self-resolving on recovery and linked to the incident when it comes true.
Response: incidents open once per cause, arrive assigned to the asset's owner with deploys and dependency status on the timeline, and alert by severity — budget-burn emergencies immediately, everything else in working hours. Learning: warnings carry their outcomes, incidents carry their evidence, and monthly SLA reports turn the stored checks into the commercial artefact. Register an asset, paste a snippet, connect a repo — the loop starts running. The deep dives linked throughout this guide are the practice; the workspace is where it runs.
Every layer of application health, and how to roll them into one honest score.
One tells you it broke; the other tells you it's breaking. That word is worth hours.
SLO vs SLA, the minutes arithmetic, a health score as budget remaining, ship or stabilise.