The AI feature is the one customers talk about, and it runs on a vendor you do not control, whose status page updates after your users have already told you. Here is how to see it first.
Any product with a feature that calls a language-model API — summarising, drafting, classifying, chatting — has a dependency that fails differently from a database or a CDN. It rarely goes fully down; it gets slow (p95 latency doubling for an hour), it rate-limits you (HTTP 429) when your traffic or theirs spikes, it returns overloaded errors (529 and similar) during incidents, it deprecates the model you pinned, it changes behaviour between versions so outputs quietly degrade, and when it does have an outage — the ChatGPT and API disruptions of 2025, the Cloudflare incidents that took AI services with them, the inference-platform errors of March 2026 — your users see your feature failing. Monitoring it means checking the dependency from outside on a schedule, measuring your own calls against a baseline, watching the vendor's status feed, and designing the feature to degrade rather than break. Facts checked 14 October 2026.
| Latency drift | p95 goes from 4 s to 12 s. Nothing errors; users abandon. Uptime check: green. |
|---|---|
| Rate limiting (429) | Your burst or their capacity. Retrying immediately makes it worse; backing off makes the feature slow. Needs a budget and a queue. |
| Overloaded (529 / 503) | Vendor-side incident. Short, frequent, and usually not on the status page until later. |
| Partial outage | One model or one region failing while others work; your check on a different model says fine. |
| Deprecation | The pinned model is retired on an announced date. Calls start failing with a clear error that nobody reads because nobody is watching. |
| Quality drift | A version change makes the JSON less reliable or the summaries longer. No error; your parser breaks downstream or the output is worse. |
| Cost spike | A prompt change or a loop multiplies tokens. Discovered on the invoice. |
| Key or quota | Key rotated, billing failed, monthly cap hit. 401 or 429 at 2 a.m. Saturday. |
Run a synthetic check against the model endpoint every few minutes: a minimal real request (a short prompt, max tokens tiny) using a dedicated monitoring key, recording HTTP status, latency and whether the response parsed. Do it for each model you depend on, not just one. Treat the result like any other API check: a baseline learned over two weeks, an early warning when latency or error rate deviates, and a confirmed incident after consecutive failures. This check is the one that tells you the vendor is degraded before their status page does — and it is the evidence when you tell your users "it is them, not us".
Pair it with the vendor's status feed (OpenAI, Anthropic and the others publish one) so that when they declare an incident, your alert carries the attribution and does not page you a second time.
Inside the app, log every model call with: model, latency, status, input and output tokens, and a feature tag. From that, four numbers per feature per hour: p50 and p95 latency, error rate split by 429 / 5xx / 529 / timeout, tokens per call, and calls per hour. Alert on deviation from each number's own baseline — the baseline-and-anomaly approach — rather than fixed thresholds, because "latency above 6 seconds" is right for one model and absurd for another. If the feature runs in the browser, frontend error capture sees the failed request and the parse error the user experiences.
Model retirement dates, key rotation dates, monthly spend caps and the vendor's rate-limit tier belong on the same calendar as certificate expiry, with alerts weeks ahead. The deprecation outage is the only one you are told about in advance; it is also the one small teams most often have anyway.
| Failure modes | Latency drift; 429 rate limits; 529/503 overload; partial outage; deprecation; quality drift; cost spike; key or quota |
|---|---|
| External check | Minimal real request per model every few minutes; status, latency, parse; dedicated key |
| Internal metrics | p50/p95 latency, error rate by status, tokens per call, calls per hour — per feature, per hour, against baseline |
| Vendor feed | Subscribe; use it to attribute and suppress, not to detect |
| Degradation | Timeouts; backoff on 429/529; fallback model or non-AI path; circuit breaker; honest message |
| Calendar | Model deprecation dates; key rotation; spend caps; rate-limit tier |
Run your own small synthetic request against the model endpoint every few minutes from outside your app, recording status and latency, and subscribe to the vendor's status feed. If your external check fails and the vendor feed shows an incident, it is them; if your check succeeds but your app's calls fail, it is your key, quota, network or code. Vendor status pages typically update after users have noticed, so your own check is the earlier signal.
Deviation from baseline in p95 latency and in error rate split by status (429, 5xx, 529, timeout), consecutive failures of your external check, a spike in tokens per call, and an approaching model deprecation or spend cap. Fixed thresholds work poorly because normal latency differs by model and prompt.
Retry with exponential backoff and a cap, queue non-urgent calls, and add a circuit breaker that stops calling after repeated failures until a check succeeds. Immediate retries during a vendor incident consume your rate budget and make recovery slower.
Calls to the retired model fail with an explicit error from the announced date. Pin model versions deliberately, record each model's retirement date on a calendar with alerts weeks ahead, and test the replacement model's output against your parsers before switching.
For a critical path, a second provider or a cached or non-AI fallback is worth the complexity; for a convenience feature, a clear 'delayed' message and a retry is usually enough. In both cases the fallback should be triggered by a circuit breaker, not by hand.
Only the complete outages. Most LLM dependency failures are latency drift, rate limiting, overload responses, partial outages by model or region, and quality drift — none of which an availability check sees. Monitoring needs a real request per model with latency and parse checks, plus metrics on your own calls.
Merik's Digital Operations monitors an LLM endpoint the way this guide describes the external check: register it as an API asset and Merik requests it every few minutes from outside, recording availability, HTTP status and response time, learning a 14-day latency and error-rate baseline per monitor, and raising at most one early warning per asset when calls drift — with a risk score, a separate confidence score and the evidence. Confirmed failures open an incident assigned to the owner and alert once by email or Slack. The OpenAI status feed is one of the vendor feeds Merik ingests, so an upstream incident annotates and suppresses the incident rather than paging you twice; the browser SDK sees the failed request and the parse error on the user's side.
Merik does not instrument your server-side calls for token counts or per-feature p95, and it does not provide fallbacks or circuit breakers — those live in your code. Merik's own AI features follow the same discipline: optional, gated, drafts only. See Digital Operations.
Free, set up for you, every module included — see what the free workspace includes
The first ten minutes, what not to do, honest communication, and shrinking the blast radius.
APIs degrade before they die — latency drift, timeouts, error trickles. Monitoring that listens.
Exposed keys, missing RLS, UI-only auth, silent dependency failures — and the afternoon monitoring setup.
Request your free company workspace and bring employees, attendance, leave, payroll and tasks together.