What To Do During an Outage: A 4-Phase Incident Response Playbook

Your startup is having its first real outage. Checkout is throwing 500s, the #alerts channel is filling with "is anyone else seeing this?", and no one on the team has run an incident before. What to do during an outage, in order: assign an Incident Commander, stop the bleeding with mitigation before you chase root cause, open one source of truth, and communicate on a fixed cadence. Recovery and the blameless postmortem come after service is restored, not during the fire. When it is your first incident, the instinct to immediately diagnose why it broke, before doing anything to stop it, is the single most expensive mistake you can make in the first ten minutes.

‍

Quick Answer: During an outage, run four phases in sequence. Phase 1, triage and containment: name an Incident Commander, declare a severity, and mitigate (roll back, reroute, scale, or toggle a flag) before investigating the cause. Phase 2, communication: open one incident bridge and post external status updates every 30 to 60 minutes for high severity. Phase 3, recovery: verify with real metrics and restore traffic gradually to avoid a thundering herd. Phase 4, postmortem: build the timeline, run a blameless 5 Whys, and assign owned, dated action items.

‍

Overview

This playbook covers:

• The first five minutes: a concrete outage checklist
• Phase 1, triage and containment, including roles and the mitigate-first rule
• Phase 2, communication, including status page best practices and customer downtime updates
• Phase 3, resolution and recovery, including the thundering herd problem
• Phase 4, the blameless postmortem, with an RCA and postmortem template
• An incident response plan template you can adopt today
• What NOT to do during an outage
• Whether an AI-native tool changes the first ten minutes
• FAQ

This is the reactive companion to knowing when you need incident management in the first place. If you are reading it mid-incident, skip to the first five minutes below. The frameworks here are not improvised. They draw on the Google SRE books, the Incident Command System (ICS) that fire and disaster response has used for decades, and ITIL v4 incident management for severity classification. Your own runbooks and architecture diagrams supply the tactical detail those frameworks leave to you.

‍

What to do during an outage: the first five minutes

Before you touch a single phase, do these five things. Speed here sets the ceiling on how fast the whole incident resolves.

1. Acknowledge the page. Someone owns it now. An unacknowledged alert is an outage nobody is working.
2. Declare an incident. Say the word "incident" out loud in your channel. Ambiguity ("might be a blip?") is what lets a SEV-1 smolder for twenty minutes.
3. Assign the Incident Commander. The first responder can self-appoint. The role matters more than the person.
4. Open one channel. A dedicated incident channel or bridge, not the general alerts firehose.
5. Post a holding update. Internal and external: "We are investigating elevated errors on checkout. Next update in 30 minutes." You have now bought yourself focus.

Everything after this is the four-phase structure, in order.

‍

Phase 1: Triage and containment (stop the bleeding)

The goal of Phase 1 is not to understand the outage. It is to end it. Restore the service baseline first, and diagnose at leisure once customers are no longer affected.

Assign roles before you assign blame

Borrowed directly from ICS and formalized by Google SRE, three roles cover most incidents:

1. Incident Commander (IC). Coordinates, decides, and holds the timeline. The IC does not touch the keyboard. Their job is to run the response, not to fix the bug.
2. Tech Lead (Operations). Drives the actual troubleshooting and mitigation. This is the only person expected to be deep in dashboards and terminals.
3. Communications Lead. Owns internal and external updates so the Tech Lead never has to stop debugging to answer "any update?" in six channels.

On a small team one person may wear two hats, but the IC and the Tech Lead should almost never be the same human. The moment your best debugger is also trying to coordinate five people, both jobs suffer. If you have not built a rotation yet, our guide to on-call setup for small teams walks through primary, secondary, and escalation paths.

The Diagnosis Trap:  The most common failure mode in the first ten minutes is reaching for root cause while the site is still down. Root cause analysis is a Phase 4 activity. During Phase 1 the only question that matters is "what action restores service right now?" A team that spends fifteen minutes reading logs to understand why a deploy broke, instead of taking ninety seconds to roll it back, has chosen understanding over their customers. Mitigate first. Diagnose later.

‍

Stop the bleeding: mitigation over root cause

Mitigation restores service. Root cause analysis explains it. During an active outage, they are not equally urgent, and treating them as if they are is what turns a ten-minute blip into a two-hour incident.

‍

Mitigation and root cause analysis answer different questions at different times: during the outage you mitigate to restore service, and only after recovery do you investigate why it failed. Confusing the two is the most common way teams extend an outage.

The standard mitigation levers, in rough order of how often they work: roll back the most recent deployment, reroute or drain traffic away from the failing component, scale up resources if the cause is load, and toggle off the feature flag behind a recent change. If a deploy went out in the last hour, roll it back before you theorize about anything else.

Assess impact and declare a severity

Assign a severity so everyone understands the stakes without a meeting. A common ITIL-aligned scale:

1. SEV-1. Critical. Core service down or major data risk, revenue actively bleeding. All hands, executive visibility.
2. SEV-2. Major. Significant degradation or a key feature down, workaround may exist. Urgent but not everyone.
3. SEV-3. Minor. Limited or cosmetic impact, no meaningful revenue or SLA exposure. Handle in business hours.

Severity drives everything downstream: who gets paged, how often you communicate, and whether you owe anyone an SLA credit.

‍

Phase 2: Communication and coordination

More incidents go badly because of poor communication than because of poor engineering. Phase 2 runs in parallel with the technical work, owned by the Communications Lead.

Establish a single source of truth

Open one dedicated incident bridge, a Slack channel or a video call, and put everything there: timestamps, actions taken, hypotheses, and decisions. The rule is simple. If it happened and it is not in the incident channel, it did not happen. This channel becomes the raw material for your postmortem timeline, so a Scribe capturing timestamps as you go saves an hour of reconstruction later.

Communicate outside the war room on a cadence

Silence reads as incompetence to everyone outside the incident. The Comms Vacuum is what forms when the responders go heads-down and stop posting: customers assume the worst, support gets buried, and executives start DMing individual engineers, which pulls them off the fix. A fixed cadence prevents it. For a high-severity incident, post an external update every 30 to 60 minutes even when the update is "still investigating, no ETA yet." Predictability is the reassurance, not the resolution.

How to communicate an outage to customers

Customer-facing updates follow a different register than the engineering channel. Lead with impact, not internals. Customers do not care that a connection pool exhausted; they care that checkout is failing and whether their data is safe. Keep each update short, state whether data is affected, and always commit to a specific next-update time.

Status page best practices

A public status page is the single highest-leverage customer communication tool during downtime. A few rules that hold across teams:

1. Host it off your own infrastructure. If your status page goes down with your app, it is worthless. Use a separate provider or domain.
2. Post before customers ask, not after. The first update should go out within minutes of declaring, even with no diagnosis.
3. Use plain, non-technical language. "Some users cannot log in," not "auth service returning 503s."
4. Commit to a next-update time and hit it. A missed promised update erodes more trust than the outage itself.
5. Always publish a Resolved update, and reference the coming postmortem for high-severity incidents.

‍

Phase 3: Resolution and recovery

An outage is not over when the fix is deployed. It is over when metrics confirm health and traffic is fully restored. Declaring victory early is how a resolved incident reopens ten minutes later.

Verify with metrics, not vibes

Confirm recovery against operational signals, not the absence of new complaints: error rates back to baseline, response latency normal, and CPU, memory, and queue depth stable. "It looks fine now" is not verification. A dashboard returning to its pre-incident shape is.

Restore traffic gradually to avoid the thundering herd

When you bring a recovered service back, do it in phases. The thundering herd problem is what happens when every retrying client, backed-up queue, and cache-cold request slams a freshly recovered service at once and knocks it straight back down. Ramp traffic in stages, keep an eye on the same metrics, and let caches and connection pools warm before returning to full load. A recovery that triggers a second outage is a common and avoidable own goal.

‍

Phase 4: Learning and the blameless postmortem

The incident is stable. Now extract the value, because an outage you do not learn from is one you have pre-purchased the right to repeat. If you are formalizing this step for the first time, our deeper walkthrough on writing a blameless postmortem covers the review meeting itself in more detail.

Build an exact timeline

Reconstruct the chronology from first alert to full resolution: when it started, when it was detected, when the IC was assigned, each mitigation attempt, and the moment of confirmed recovery. If your Scribe captured timestamps live in the incident channel, this is transcription rather than archaeology.

Run a blameless root cause analysis

Blameless does not mean nobody is accountable. It means you attack the systemic conditions that let a human error become an outage, not the human. The premise, straight from the Google SRE workbook: if a single engineer's mistake can take down production, the system that allowed it is the defect, not the engineer.

The 5 Whys is the workhorse technique. Ask "why" until you reach a systemic cause rather than a person:

Problem: Checkout returned 500s for 54 minutes.
1. Why? A deploy introduced a null-pointer path in the payment handler.
2. Why did it reach production? Tests did not cover the null case.
3. Why not? The test suite has no integration coverage for the payment path.
4. Why not? Payment tests were deprioritized under deadline pressure.
5. Why? There is no policy requiring integration coverage for revenue-critical paths.

Root cause (systemic): No coverage requirement for revenue-critical
code paths. Fix: gate deploys on payment-path integration tests.

‍

Notice the answer at the bottom is a missing guardrail, not a careless engineer. That is the blameless move.

Write the postmortem

A postmortem is the durable artifact. A workable template:

# Postmortem: [Incident title]
Date: [date]   Severity: [SEV-x]   Duration: [start to resolution]
Author: [name]   Status: [draft / reviewed / actioned]

## Summary
2-3 sentences: what broke, who was affected, how long, how it ended.

## Impact
Customer, revenue, and SLA impact in concrete numbers.

## Timeline (all times UTC)
HH:MM  First alert
HH:MM  Incident declared, IC assigned
HH:MM  Mitigation attempted: [action]
HH:MM  Service restored
HH:MM  Incident closed

## Root cause
The systemic cause, per the 5 Whys. Blameless.

## What went well / what went poorly
Detection speed, comms, tooling gaps. Be honest.

## Action items
| Action | Owner | Due date | Ticket |
|---|---|---|---|
| ... | ... | ... | ... |

‍

The action items table is the only part that prevents recurrence. A postmortem with a beautiful root cause and no owned, dated follow-ups is a diary entry, not a fix.

Assign actionable, owned follow-ups

Every action item needs an explicit owner and a deadline, tracked in your normal ticketing system, not buried in the doc. "We should improve test coverage" is not an action item. "Add payment-path integration tests, owner: Priya, due: Oct 3, JIRA-4821" is.

‍

Incident response plan template

If your team does not have a plan yet, this is a complete starting point you can paste into a runbook today.

# Incident Response Plan

## 1. Declare
Anyone can declare an incident. Say "declaring an incident" in
#incidents. When in doubt, declare.

## 2. Roles
- Incident Commander: coordinates, owns timeline, does not code.
- Tech Lead: drives mitigation and diagnosis.
- Comms Lead: internal + external updates.

## 3. Severity
- SEV-1: core service down / revenue or data at risk. All hands.
- SEV-2: major degradation, workaround may exist. Urgent.
- SEV-3: minor / cosmetic. Business hours.

## 4. Mitigate first
Restore service before finding root cause. Levers: roll back,
reroute, scale, toggle flag.

## 5. Communicate
- One incident channel = single source of truth.
- External status update every 30-60 min for SEV-1/2.

## 6. Recover
Verify with metrics. Restore traffic gradually.

## 7. Learn
Blameless postmortem within 5 business days. 5 Whys.
Owned, dated action items.

‍

Adapt the severities and cadences to your exposure, but keep the order. The sequence, declare, mitigate, communicate, recover, learn, is what turns panic into a process.

‍

What NOT to do during an outage

Knowing the anti-patterns is as useful as knowing the steps. Avoid these:

1. Do not hunt for root cause while the site is down. Mitigate first. This is the Diagnosis Trap, and it is the most expensive mistake in incident response.
2. Do not go silent. No external update for an hour during a SEV-1 does more reputational damage than the outage. Communicate even when you have nothing new.
3. Do not let everyone debug at once. Five engineers making uncoordinated changes to a live system create a second incident. The IC serializes the response.
4. Do not declare victory on a hunch. Verify recovery with metrics before you post Resolved and before you restore full traffic.
5. Do not run a blameful postmortem. The moment a postmortem names a person as the cause, people stop reporting incidents honestly, and your data goes dark. Attack the system.
6. Do not skip the postmortem because the fix was "obvious." Obvious fixes to symptoms leave systemic causes in place. That is how the same outage returns in a month.

‍

Does an AI-native tool change the first ten minutes?

The four phases are the same whether a human or a machine kicks them off. What changes is where the response starts. In a traditional setup, a human is paged cold and has to answer "what broke?" from a blank page while the clock runs. Tools that add an AI investigation layer, such as Vibe OnCall, invert that.

A traditional pager wakes a human and hands them a mystery; an AI-native tool investigates the alert before paging, so Phase 1 starts with context instead of a cold search. One mid-market customer reported a 60% MTTR reduction after adopting Vibe OnCall's pre-paging triage. The playbook does not change, but the starting line moves forward.

Whichever path you are on, the discipline is identical: mitigate before you diagnose, communicate on a cadence, verify before you celebrate, and learn without blame.

‍

Frequently asked questions

What is the first thing to do during an outage?

‍Acknowledge the alert and declare an incident, then assign an Incident Commander. The IC owns coordination and the timeline while a separate Tech Lead drives the fix. Naming ownership in the first two minutes is what prevents an alert from smoldering unworked.

Should you find the root cause during an outage?

‍No. During an active outage, mitigate to restore service first (roll back, reroute, scale, or toggle a flag). Root cause analysis is a post-incident activity for the blameless postmortem. Chasing the cause while the service is down extends the outage and is the most common early mistake teams make.

How often should you update customers during downtime?

‍Every 30 to 60 minutes for a high-severity incident, even when there is no new information. Post the first update within minutes of declaring, keep it in plain non-technical language, and always commit to a specific next-update time. Predictable communication rebuilds trust faster than the fix alone.

What is a blameless postmortem?

‍A blameless postmortem investigates the systemic conditions that allowed an incident, rather than assigning fault to an individual. The principle, from the Google SRE workbook, is that if one person's error can cause an outage, the system is the defect. Blameless reviews keep engineers reporting incidents honestly, which keeps your reliability data accurate.

What is the thundering herd problem during recovery?

‍The thundering herd problem is when a recovered service is immediately overwhelmed by retrying clients, backed-up queues, and cold caches all hitting it at once, knocking it back down. Prevent it by restoring traffic gradually in stages and watching your metrics as you ramp, rather than flipping everything back to full load.

What should an incident response plan include?

‍At minimum: how to declare an incident, defined roles (Incident Commander, Tech Lead, Comms Lead), a severity scale (SEV-1 to SEV-3), a mitigate-first rule, a communication cadence, recovery verification, and a blameless postmortem step with owned action items. The order matters as much as the contents: declare, mitigate, communicate, recover, learn.

‍