Incidents

Did the deploy break production? Correlating deployments with incidents

The first question in any incident is "what changed?" — and in most small teams the answer is a Slack search for "deployed". Put the deploy on the timeline and the question answers itself.

An incident timeline with a deploy marker minutes before the error spike

The majority of production incidents follow a change, and the fastest way to answer "was it the deploy?" is to have every deployment recorded on the same timeline as the monitoring — automatically, from the deploy tool's webhook — so an incident opening nine minutes after a deploy shows the deploy beside it. Correlation is not proof: a deploy that coincides with a provider outage is innocent, and a deploy whose effect takes an hour to surface will look innocent when it is not. But a deploy marker beside an error spike turns a thirty-minute investigation into a two-minute rollback decision, and the habits that go with it — small deploys, a ten-minute watch window after each one — make most of those decisions unnecessary.

Key takeaways
  • Most incidents follow a change. If deploys are not on your monitoring timeline, the first ten minutes of every incident is spent finding out what shipped.
  • Record deploys automatically from webhooks — GitHub, Vercel, your CI — never by asking people to post in a channel.
  • Read correlation honestly: a deploy nine minutes before an error spike is a strong lead, not a verdict. Check for provider incidents and traffic changes too.
  • Rollback first, investigate second when the correlation is tight and rollback is cheap. Investigate first when rollback is risky or the effect is delayed.
  • The habit that makes this rare: small deploys and a ten-minute watch after each one, by the person who shipped it.

Why "what changed?" is the first question

Systems that were working and are now not working usually had something done to them: a deploy, a config change, a dependency update, a certificate rotation, a scaling event, a provider change. Of these, deploys are the most frequent and the most reversible. An incident that starts within minutes of a deploy has a prime suspect and a cheap remedy. An incident with no deploy nearby needs a different investigation — a provider, a traffic pattern, a slow leak. Knowing which situation you are in, in the first minute, is worth more than any dashboard. Most incident time is spent not knowing.

Getting deploys onto the timeline

The wrong way is a convention — "post in #deploys when you ship" — which is followed for a month. The right way is a webhook from wherever deploys happen: GitHub (deployment or push events), Vercel (deployment succeeded), your CI pipeline's final step. The webhook records what was deployed, to which asset, when, by whom, and the commit or version. That record sits on the same timeline as checks, warnings and incidents, so an incident view shows "deploy v2.14.1 at 13:58 · errors rising from 14:02 · incident opened 14:04". No one had to remember anything.

For agencies, the same applies per client site: the WordPress plugin update or the theme deploy on a client's site is a change, and it belongs on that client's timeline. Per-client monitoring.

Reading the correlation honestly

Tight correlation, nothing else changedDeploy at 13:58, errors from 14:02, no provider incident, normal traffic. Probably the deploy. Roll back.
Tight correlation, provider also downDeploy at 13:58, Cloudflare reporting an outage from 13:55. Probably not the deploy. Do not roll back; wait. Is it us or them?
Loose correlationDeploy at 11:00, errors from 14:00. Possibly the deploy — a slow leak, a cache expiry, a cron that runs at 14:00 — but investigate before rolling back three hours of other work.
No deploy nearbyLook elsewhere: traffic, dependency, certificate, data. The ten issues monitoring should detect.

The principle: a deploy marker is context, not an accusation. Monitoring that says "the deploy caused it" is overclaiming; monitoring that says "a deploy landed nine minutes before this started" is doing its job.

Rollback or investigate?

The habits that make the question easy

  1. Small deploys. A deploy with three changes has three suspects; a deploy with forty has forty. Ship more often, ship less each time.
  2. The ten-minute watch. The person who deployed watches the asset's error rate and latency against baseline for ten minutes. Most deploy-caused incidents show in the first five.
  3. Deploy in daylight. Not at 6pm Friday; not during a provider incident; not during the client's peak hour.
  4. Track the change failure rate. Deploys that led to an incident ÷ deploys. Above 15% and the deploys are too big or the pre-deploy checks too thin. Cutting the chain early.

Quick reference

First questionWhat changed? — answered by deploys on the monitoring timeline
SourceWebhooks from GitHub, Vercel or CI; never a manual post
Correlation ruleContext, not accusation — check providers and traffic before concluding
Roll back first whenTight correlation, cheap rollback, no migration
Investigate first whenDelayed effect, risky rollback, provider also reporting trouble
HabitsSmall deploys · 10-minute watch · daylight · change failure rate under 15%

Frequently asked questions

How do I know if a deployment caused a production incident?

Put every deployment on the same timeline as your monitoring, automatically from the deploy tool's webhook, and look at what preceded the incident. A deploy a few minutes before an error spike, with no provider incident and normal traffic, is a strong lead and usually justifies a rollback. A deploy hours earlier, or one that coincides with a provider outage, needs investigation before you conclude.

How do I track deployments alongside monitoring?

Use a webhook from wherever deploys happen — GitHub deployment or push events, Vercel deployment events, or your CI pipeline's final step — to record what was deployed, to which asset, when and by whom, on the monitoring timeline. Asking people to post in a channel works for about a month; the webhook works permanently.

Should I roll back immediately when errors rise after a deploy?

Roll back first when the correlation is tight, rollback is a single command and the deploy did not run a data migration; investigate afterwards on the restored system. Investigate first when a migration ran, when the effect is delayed, or when a provider is also reporting trouble — rolling back in those cases can make things worse.

What is change failure rate?

The fraction of deployments that led to a degradation or incident requiring remediation. Small teams that track it usually aim for under 15%; above that, deploys are typically too large or pre-deploy checks too thin. It is computed from the same records — deploys and incidents on one timeline.

What should I watch after a deployment?

The deployed asset's error rate and latency against its own baseline, and frontend errors from real users, for about ten minutes, by the person who shipped. Most deploy-caused problems appear within the first five minutes; a warning during that window is the cheapest incident you will ever have.

Can monitoring tell me that a deploy caused an incident?

It can tell you a deploy landed shortly before the incident, which is context rather than proof. Good monitoring presents the deploy beside the incident with the timing and lets a person conclude; monitoring that declares the deploy the cause is overclaiming, because provider outages and traffic changes coincide with deploys too.

How Merik handles it

Merik's Digital Operations module takes deploys from GitHub and Vercel webhooks and places them on the incident timeline of the asset they touched. When an early warning or incident opens, a deploy that landed just before is shown as correlated context — with the time gap — never as a verdict; if a provider is also reporting trouble, that appears alongside from the vendor status feeds, so the "us or them" and "was it the deploy" questions are answered by the same view.

Because each asset's latency and error-rate baselines are measured over 14 days, the ten-minute post-deploy watch is a glance at one screen: current against normal. Deploys and incidents on one timeline also give the monthly per-client SLA report its "changes" section for free. See the Digital Operations module.

Create your workspace →

Or talk to us about your team →