Total outages announce themselves. The expensive backend failures are quieter: the process stays up, most things work, and something important has stopped. Here is how those happen — and how they get caught.
The most damaging backend errors are partial failures: the application keeps running while one route returns 500s, one dependency times out, one background job stops, or one response goes subtly wrong. Nothing crashes, so nothing restarts and no "is it up?" check complains. Detecting this class requires watching behaviour per endpoint from the outside — status codes, latency against baseline, error-rate deviation — because the process-level signals that ops tooling traditionally watches all remain green while it happens.
The mental model of backend failure — the server "goes down" — describes the rarest case. Modern backends are supervised: crashed processes restart, failed health checks pull instances from the pool, orchestrators replace what dies. Total failure is handled by the platform.
What the platform does not handle is working incorrectly. A route handler that throws on a specific input returns a 500 — the process is fine. A connection pool sized for last year's traffic queues requests under load — everything completes, slowly, until the queue itself times out. A third-party API degrades and every request through it inherits the degradation. A queue consumer dies while the queue keeps accepting — producers succeed, work silently stops.
In every case the process answers its liveness probe, CPU looks normal, and the aggregate error rate barely moves. The failure is real, user-visible, and invisible to process-level monitoring — by construction.
You cannot see inside every handler, but every quiet failure mode above leaks exactly one of four external symptoms. Watch all four, per endpoint:
Two supporting correlations sharpen all four: deploy events (the failing route usually started failing at a deploy) and vendor status feeds (the degraded dependency is often publicly admitting it). A monitoring system holding both timelines can say "500s on /api/orders began 4 minutes after deploy a1b2c3" or "checkout degradation coincides with the payment provider's incident" — which converts detection into half a diagnosis.
| The dominant failure mode | Partial: one route, one dependency, one consumer — while the process stays healthy |
|---|---|
| Why platforms miss it | Supervisors handle crashed processes, not incorrect behaviour; liveness probes stay green |
| The four external symptoms | Error status codes, timeouts, latency deviation from baseline, wrong response content |
| Per-endpoint rule | Monitor money/journey endpoints individually — aggregates hide the dead route |
| Timeout meaning | Usually the first external sign of resource exhaustion (pools, queues) building underneath |
| Sharpening correlations | Deploy events on the same timeline; vendor status feeds for inherited degradation |
A partial failure is when an application keeps running but some subset of its behaviour is broken: one route returning errors, one dependency timing out, one background consumer stopped, or responses that are wrong despite success status codes. The process passes health checks throughout, which is why partial failures evade process-level monitoring.
Because most backend monitoring watches the process — is it up, is CPU normal — while partial failures leave the process healthy. The failure is only visible behaviourally: a specific endpoint's status codes, its latency against its own history, its timeout rate. Teams monitoring only aggregates or only liveness discover these failures from users.
Outside-in behavioural monitoring covers the major quiet failure modes: per-endpoint checks catching error status codes and asserting expected content, latency tracked against a measured baseline to expose gradual degradation, timeout rates tracked separately, and deploy plus vendor-status timelines for correlation. Agent-based telemetry adds depth later, but the outside-in layer is what tells you something is wrong at all.
Because they carry different information. An error status is the server answering "no"; a timeout is the server failing to answer at all, which typically means requests are stalling inside it — the classic signature of connection-pool or queue exhaustion. A rising timeout rate often precedes a full outage by enough time to act, but only if it is visible as its own signal.
Content assertions: for critical endpoints, the monitoring check verifies the response contains what a correct answer must contain, not just an acceptable status code. Frontend error monitoring provides a second net — wrong shapes returned to the browser often throw client-side errors that browser telemetry reports even though the server logged success.
Expose the queue's state where a check can see it — a small status endpoint reporting queue depth and the age of the last completed job — and monitor it like any other endpoint. Producers succeeding while consumers are dead is otherwise one of the longest-lived silent failures, because nothing in the request path ever errors.
Merik watches backends the way this article recommends: from the outside, per endpoint, against history. Register the endpoints that matter as monitored assets and each gets its own checks — status validation, optional content assertions, latency recorded into a 14-day baseline. A 500 is confirmed before it becomes an incident; a timeout is a tracked failure stage of its own; a latency climb against baseline raises an early warning while everything still technically works.
The correlations ship too: GitHub and Vercel webhooks put deploys on the incident timeline, and live vendor status feeds mean an endpoint failing because a payment or hosting provider is down says so — recorded, attributed, and not paged as if it were your bug. Merik does not install agents inside your processes; it makes the outside view sharp enough that the quiet failures stop being quiet.
APIs degrade before they die — latency drift, timeouts, error trickles. Monitoring that listens.
The failures nobody notices are the ones that compound. What silence costs, and how to end it.
Logs are records, not lookouts — nobody reads them until after the incident.