Incident Management Architecture: Severity Levels, On-Call, Blameless Postmortems and Error Budgets (2026)

Most teams do not have an incident problem; they have an incident-system problem. The alert fires, someone wakes up, five people join a call, three of them change production at once, nobody writes anything down, and the postmortem a week later blames “human error”. The outage was survivable. The way the organisation handled it was not repeatable.

Effective incident management is an architecture, not a heroic act. It has components (detection, severity classification, command roles, communication channels, runbooks, a learning loop) and each component has a measurable contract with the next. When those contracts are explicit, a 3 a.m. page becomes a rehearsed procedure instead of an improvisation, and the error budget turns every incident into a priced input to roadmap decisions.

This article lays out that architecture end to end: a lifecycle model, a severity matrix you can adapt, the incident command structure from Google’s SRE books, burn-rate alerting with worked arithmetic and runnable Prometheus rules, sustainable on-call design, postmortem mechanics, and the metric traps (especially MTTR) that quietly corrupt the whole loop.

What this covers: the incident lifecycle, severity design, command roles, alert quality with multi-window burn rates, on-call load limits, communication and runbooks, postmortems and action tracking, error-budget policy, metrics pitfalls, tooling, and failure modes.

Context and Background

Modern incident practice descends from two traditions. The first is the Incident Command System used by emergency services, adapted by Google and others into a software form with a small set of explicit roles. The second is the site reliability engineering (SRE) discipline, which reframed reliability as a quantified target (a service level objective, or SLO) with a finite allowance for failure (the error budget). Google’s public SRE book chapters on managing incidents, being on-call and postmortem culture remain the most cited primary sources, and the later SRE Workbook added concrete alerting math and an example error-budget policy.

Vendors then productised the practice. PagerDuty publishes an open incident response guide, including a severity scale from SEV-1 to SEV-5. Atlassian, Incident.io, FireHydrant and Grafana OnCall each ship tooling that wraps paging, a chat-based incident channel, status pages and postmortem templates. The tools differ, but they converge on the same skeleton, which is why it is worth designing the skeleton first and picking tools second.

What changed by 2026 is mostly the surrounding load. Systems are more distributed, so a single user-visible symptom has more possible causes. Telemetry is richer thanks to OpenTelemetry (see our guide to W3C Trace Context and distributed tracing propagation), but richer telemetry also means more alert surface and more ways to page a human for something that did not matter. AI-assisted triage is now common enough to deserve its own treatment, which we cover in AIOps incident response and agentic SRE. This article is the human-process foundation those tools sit on.

The thesis here is a simple one that most top-ranking guides do not state plainly: the unit of design is the feedback loop, not the response. A team that responds brilliantly but never converts incidents into funded reliability work will have the same outage again. A team with a mediocre response but a tight loop (alert quality feeds severity, severity feeds postmortem triggers, postmortems feed budget policy, budget policy feeds the roadmap) improves every quarter. The rest of this article is organised around closing that loop.

A quick vocabulary note, since terms are used loosely. An event is anything observable. An alert is an event someone chose to notify about. An incident is a situation that needs coordinated response because it threatens an objective. A major incident is one that crosses a severity threshold and activates the full command structure. Keeping these four separate matters later, when we discuss alert-to-incident ratios and when to declare.

The Incident Management Lifecycle and Reference Architecture

The incident lifecycle has six stages: detect, triage, declare, mitigate, resolve and learn. Each stage has one owner, one input and one exit criterion. Mitigation (stopping user harm) is deliberately separated from resolution (fixing the cause), because the fastest path to a healthy service is usually a rollback or failover, not a diagnosis.

Figure 1: The incident management lifecycle. The dashed edge is the point of the whole architecture: tracked fixes change what the next detection looks like.

Figure 1 shows the stages as a loop. Read it as a set of contracts. Detection promises that a user-visible problem produces a page within minutes, without false pages. Triage promises a severity within a few minutes of acknowledgement. Declaration promises a named incident commander (IC) and a single coordination channel. Mitigation promises that one role changes production at a time. Resolution promises that recovery is confirmed against the same service level indicator (SLI) that detected the problem. Learning promises a written, reviewed record with owned action items.

Detect: symptoms over causes

The Google SRE guidance on monitoring is blunt: page on symptoms, not causes. A symptom is something a user would notice, such as elevated error ratio or latency at the edge. A cause is an internal fact such as high CPU on a node or a full queue. Causes belong on dashboards and in tickets; they page only when they are a reliable and imminent predictor of a symptom. Every cause-based page you remove reduces on-call load without reducing coverage, because the symptom alert will still fire if the cause matters.

The practical test for any page is three questions. Is a human action required right now? Is the action something a machine cannot do? Is user-visible harm either happening or provably imminent? A “no” to any of them means the signal should be a ticket, a dashboard annotation or an automated remediation, not a page.

Triage and declare: decide early, argue later

Triage assigns a severity and decides whether to declare an incident. The Google SRE Workbook’s incident response chapter is explicit that teams should declare early and often, and that declaration criteria should be agreed in advance by reviewing past outages. The reason is asymmetric cost. An unnecessary declaration costs a few people a few minutes. A late declaration costs an extended outage with unclear ownership, which the Workbook illustrates with a case where delayed declaration led to miscommunication and a weekend-long disruption.

PagerDuty’s open response guide offers the matching rule for severity: if you are unsure between two levels, treat the incident as the higher one, and review the classification in the postmortem rather than debating it mid-incident. Both rules push the same direction: take the cheap precaution now and fix the paperwork later.

Mitigate, resolve, learn

Mitigation options form a short ordered list that should live in your runbooks: roll back the most recent change, shift traffic away from the failing region or cell, shed load, disable a feature flag, scale out, and only then debug forward. The ordering reflects blast radius and reversibility. A rollback is reversible and well-rehearsed; a hotfix written at 3 a.m. is neither.

Resolution means the SLI that triggered detection has recovered and stayed recovered for a defined soak period. Closing the incident is an explicit act by the IC. The Workbook notes that if the IC does not formally close an incident, people assume it is still ongoing, which wastes attention and muddies the timeline used later for the postmortem.

Learning is the stage most organisations under-invest in, and it is where the architecture earns its keep. We return to it in detail below. For now, note the structural point: learning is a stage with an owner and an exit criterion (reviewed postmortem, action items entered in the tracker), not an optional follow-up.

The reference architecture in components

Behind the lifecycle sits a small set of components. Telemetry and SLI recording feed an alerting layer, which routes to a paging service with escalation policies. The paging service opens a coordination space (a chat channel and, for bigger incidents, a call bridge) and a timeline document. A status page and a stakeholder update channel carry outward communication. A runbook repository, linked from every alert, carries the known mitigations. A postmortem repository and an action tracker close the loop, and an error-budget calculation ties the history back to release policy.

None of these components is exotic. What distinguishes mature setups is that the links between them are automated or at least mandatory: every page carries a runbook URL, every declared incident auto-creates a channel and a timeline, every closed incident of sufficient severity auto-creates a postmortem ticket with a due date. Automation here is not about speed; it is about removing the decisions a stressed human will otherwise skip.

Designing Severity Levels That People Actually Use

Severity levels exist to answer three questions quickly: who gets woken up, how much process to run, and who must be told. A good scale is small, metric-driven and bound to actions. A bad scale is a poetry contest (“critical”, “major”, “serious”) that every responder reads differently at 3 a.m.

PagerDuty’s public guide uses five levels. In summary: SEV-1 is a critical issue that needs public notice and executive liaison; SEV-2 severely hinders many customers and still triggers major-incident response with an IC paged; SEV-3 covers minor customer impact needing immediate attention from the owning team; SEV-4 is a minor issue that does not affect customers’ ability to use the product; and SEV-5 is cosmetic and becomes a ticket. Anything above the SEV-3 line counts as a major incident. The guide also recommends that you replace generic wording with your own definitions, usually expressed as a percentage of users or accounts affected.

A severity matrix you can adapt

The matrix below is a starting point, not a standard. The percentages and durations are illustrative; calibrate them against your own traffic and your SLOs. The structural idea is that every row binds an impact definition to a response, a communication duty and a postmortem requirement.

Severity	Impact definition (illustrative)	Who is paged	Command structure	Communication duty	Postmortem
SEV-1	Core user journey down or data loss or security exposure, over 25% of users	On-call plus IC plus leadership liaison	Full: IC, Ops, Comms, Planning	Status page within 15 min, updates every 30 min	Mandatory, reviewed
SEV-2	Core journey degraded for a large minority, SLO fast burn firing	On-call plus IC	IC, Ops, Comms	Status page within 30 min, updates hourly	Mandatory
SEV-3	Partial loss for a small segment or a redundancy gap	Service on-call only	On-call acts as IC	Channel update, stakeholder ping	Optional, short form
SEV-4	No user impact, action needed this week	Ticket or low-urgency page	None	None	Not required
SEV-5	Cosmetic	Ticket	None	None	None

Three design rules make a matrix like this survive contact with reality.

Tie severity to user impact, not to technical cause. “Database primary failed” is not a severity; “checkout success rate below 95%” is. The cause may be dramatic and the impact negligible (a replica fails and nothing changes), or the reverse (a silent bug corrupts a small fraction of writes). Cause-based scales misrank both.

Make severity changeable in both directions without ceremony. An incident can be upgraded when impact widens and downgraded when mitigation lands. Record each change with a timestamp, because the sequence tells the postmortem reader what the team knew and when.

Bind severity to the error budget. If your SLO tooling can tell you that a given outage consumed X% of the monthly budget, you can let that number inform classification. The Google SRE Workbook’s example error-budget policy, discussed later, uses exactly that sort of threshold: a single incident that consumes more than 20% of the four-week budget forces a postmortem with at least one top-priority action item. Wiring budget consumption into severity makes the scale self-calibrating.

Priority versus severity

Teams often conflate severity (how bad is the impact) with priority (how soon will we fix the underlying defect). They diverge in practice. A SEV-1 caused by a one-off certificate expiry can be mitigated in minutes, and the permanent fix (automated renewal) might be a medium-priority item. A SEV-4 that is a slow memory leak may be top priority this sprint because it will become a SEV-2 on Friday. Keep two fields. Severity drives the live response; priority drives the follow-up queue.

Command Structure: Roles, Channels and Handoffs

Incident command separates three activities that stressed engineers naturally blend: thinking about the whole incident, changing the system, and talking to people. Google’s SRE book describes four roles. Incident Command keeps the high-level picture, assigns responsibilities and holds any role it has not delegated. Operational Work applies tools to the problem, and only this group should modify the system during an incident. Communication is the public face of the response team. Planning handles longer-term needs such as filing bugs, arranging handoffs and tracking how the system has drifted from normal so it can be reverted afterwards.

Figure 2: Incident command roles. Only the Operations side touches production; the commander delegates and removes obstacles but does not debug.

The single most valuable rule in the whole model is that only one group modifies production. Google’s own illustration of an unmanaged outage hinges on a well-meaning colleague making an uncoordinated change that worsens the incident. When two people “just try something” in parallel, the timeline becomes unreadable, mitigations cancel each other and the root cause is lost. Funnelling changes through an Ops lead is cheap insurance.

Role mechanics that matter in practice

The IC should not be the deepest technical expert in the room. If the IC is debugging, nobody is coordinating. In small incidents the on-call engineer holds all roles by default, which the SRE Workbook acknowledges: until roles are delegated, the IC holds them all, and the communications role can fold back into the IC for a small incident. The skill is recognising when to stop doing and start delegating, and to hand command to someone else when the workload is too heavy.

A scribe is not formally one of the four classic roles in the SRE book, but nearly every operating team adds one, usually under Planning or Communications. A timestamped log of observations, hypotheses, actions and decisions is the raw material of the postmortem. Without it, reconstruction depends on memory, which is least reliable exactly when the stakes are highest.

The Workbook recommends moving detailed debugging out of chat and into a shared document once three or more people are involved, keeping the chat channel for decisions and status. This addresses a common failure: a channel with 40 people and 600 messages, where the one critical observation scrolled away in minute eight.

Handoffs and rotation on long incidents

Long incidents exhaust people. The Workbook describes rotating the IC and responders so people can rest and fresh perspectives arrive, and cites a case where an eight-hour incident was staffed with four-hour rotations. A handoff should be explicit and verbal (“you are now IC”, acknowledged), include a current-state summary, and be logged. Command can also move upward to a more experienced responder mid-incident; the Workbook’s GKE example has the on-call engineer confirming customer impact, following the documented declaration procedure, then handing command to a senior SRE who set up the formal structure. Handing off command is not failure. It is the process working.

Practising before it counts

The roles only work if people have used them. Two low-cost practices appear throughout the SRE literature. The first is the Wheel of Misfortune: a new engineer plays the on-call role in a reenactment of a past incident while a facilitator, often the original IC, plays the rest of the world. The second is routine use, meaning you run the incident process for minor events too, so the vocabulary and channel setup are muscle memory. The SRE book’s best-practice list includes both “practice” and “rotate roles” for exactly this reason. Chaos experiments make a natural rehearsal venue: our guide on chaos engineering with Chaos Mesh and Litmus shows how to inject controlled failures with a defined steady state, which gives responders a safe incident to command.

Communication architecture

Communication has two audiences with different needs. Responders need a single place to coordinate and a shared picture of current state. Everyone else needs a predictable cadence of short, honest updates. The Workbook’s advice is to agree the channel in advance so the IC does not have to choose one mid-incident, to keep contact lists and message templates prepared, and to send regular updates because silence leads people to assume nothing is being done.

A usable update template has five fields: current impact in user terms, what we know, what we are doing, what we do not know yet, and the time of the next update. Committing to the next-update time matters more than the content, because it converts anxious polling into a schedule. Internally, post to a dedicated stakeholder channel; externally, use a status page with plain language and no internal jargon. Never speculate on root cause in a public update. “We have identified the affected component and are rolling back a recent change” is accurate and sufficient.

Alert Quality: Multi-Window Burn Rates With Worked Math

The direct answer: alert on how fast you are spending the error budget, not on raw thresholds. A burn rate of 1 consumes exactly the whole budget over the SLO period. A burn rate of 14.4 on a 30-day, 99.9% SLO exhausts 2% of the budget in one hour. Pairing a long window with a short window pages quickly on real incidents and stops paging soon after they end.

For a fuller treatment of SLO definition and tooling, see our guide to SLOs, error budgets and burn-rate alerting for IoT platforms with OpenSLO and Sloth. Here we focus on how the alerts plug into incident management.

The arithmetic

Start with a 99.9% availability SLO over 30 days. The error budget is 0.1% of requests. If you measure in time and assume a full outage, 30 days is 43,200 minutes, so the budget is 43.2 minutes of total downtime. In request terms, if the service handles 10 million requests a month (illustrative), the budget is 10,000 failed requests.

Burn rate is the observed error ratio divided by the budgeted error ratio. With a 0.1% budget ratio, an observed 1.44% error ratio is a burn rate of 14.4. The Google SRE Workbook gives the general relation for the share of budget consumed by the time an alert fires: budget consumed equals burn rate times alert window, divided by the SLO period. For the fast-burn case, 14.4 times one hour over 720 hours is 0.02, or 2% of the monthly budget.

Time-to-exhaustion follows directly. At burn rate 14.4, the whole budget is gone in 720 / 14.4 = 50 hours. At burn rate 6, it takes 720 / 6 = 120 hours, five days. At burn rate 1, it takes the full 30 days. Those three speeds correspond to three different human responses: stop everything, deal with it today, and plan it into the week.

The recommended alert tiers

The Workbook’s recommended parameters for a 99.9% SLO are reproduced below, with the observed error ratio that triggers each (burn rate times the 0.1% budget ratio).

Severity	Long window	Short window	Burn rate	Error ratio threshold	Budget consumed at firing	Time to exhaust
Page	1 hour	5 minutes	14.4	1.44%	2%	50 hours
Page	6 hours	30 minutes	6	0.6%	5%	5 days
Ticket	3 days	6 hours	1	0.1%	10%	30 days

Check the middle row with the formula: 6 times 6 hours over 720 hours is 0.05, so 5%. The ticket row: 1 times 72 hours over 720 hours is 0.10, so 10%. The Workbook also gives a guideline for the short window: make it one twelfth of the long window. That is why 1 hour pairs with 5 minutes, 6 hours with 30 minutes, and 3 days with 6 hours.

Why two windows? The long window gives statistical confidence and a meaningful budget impact. The short window confirms the burn is still happening, which lets the alert reset quickly once errors stop. A single 1-hour window alone would keep firing for up to an hour after the problem was fixed, pointing a responder at a ghost. Requiring both windows to exceed the threshold removes that tail.

Runnable Prometheus rules

The following sketch implements the fast-burn page and the slow-burn ticket for an HTTP service, assuming a counter http_requests_total with a code label and a service label. It uses recording rules for each window so the alert expressions stay readable and cheap. Adjust the metric and label names to your instrumentation. The rule-file structure (groups, rules, record, alert, expr, for, labels, annotations) follows the Prometheus documentation.

groups:
  - name: slo-recording-checkout
    rules:
      - record: service:error_ratio:rate5m
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[5m]))
      - record: service:error_ratio:rate1h
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[1h]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[1h]))
      - record: service:error_ratio:rate6h
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[6h]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[6h]))
      - record: service:error_ratio:rate3d
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[3d]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[3d]))

  - name: slo-alerts-checkout
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          service:error_ratio:rate1h{service="checkout"} > (14.4 * 0.001)
          and
          service:error_ratio:rate5m{service="checkout"} > (14.4 * 0.001)
        labels:
          severity: page
          slo: checkout-availability-99.9
        annotations:
          summary: "Checkout is burning its 99.9% SLO error budget at 14.4x"
          runbook_url: "https://runbooks.example.com/checkout/availability"
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          service:error_ratio:rate3d{service="checkout"} > (1 * 0.001)
          and
          service:error_ratio:rate6h{service="checkout"} > (1 * 0.001)
        labels:
          severity: ticket
          slo: checkout-availability-99.9
        annotations:
          summary: "Checkout is consuming error budget at the sustainable limit or faster"
          runbook_url: "https://runbooks.example.com/checkout/availability"

Three practical notes. First, include the 6-hour and 30-minute medium-burn pair by copying the fast-burn pattern with 6 times 0.001 as the threshold; it is omitted here for length. Second, the expression must handle a zero-traffic denominator, because a division by zero yields NaN and the comparison is false, which silently means “no alert”. For low-traffic services, add a minimum-traffic guard or an absent-metric alert. Third, keep runbook_url mandatory in every paging rule; a lint step in CI that rejects a paging rule without it costs nothing and pays back every night.

Figure 3: The alert pipeline. Three burn-rate tiers split into a page path and a ticket path, so only urgent budget burn wakes a human.

Routing and inhibition

Alertmanager adds the routing layer. Group alerts by service and severity so one failure produces one notification, not forty. Use inhibition so that a broad outage alert suppresses narrower child alerts: when the regional load balancer is down, per-service latency alerts add noise, not information. Route severity: page to the paging service and severity: ticket to the work queue. The goal the SRE book states for sustainable operations is an alert-to-incident ratio approaching one to one, where each page corresponds to a real incident needing action. A ratio far above that means the on-call is triaging noise.

Where burn-rate alerts are weak

Burn-rate alerting is strong on user-visible, high-volume services and weaker elsewhere. Low-traffic services produce noisy ratios, where a single failed request out of ten is a 10% error ratio. Batch pipelines and data freshness need different SLIs (age of newest data, job success within a deadline). Multi-tenant systems can hide a total outage for one customer inside a healthy global ratio. For those, add per-segment SLOs for your top tenants, or synthetic probes that exercise critical journeys independent of organic traffic. The architecture should name these gaps explicitly rather than pretend one alert type covers everything.

Building Sustainable On-Call

On-call is the human actuator of the whole system, and it is the part most likely to be quietly destroyed by over-use. The Google SRE book gives concrete anchors worth adopting as starting points. SREs should spend at least 50% of their time on engineering work; of the remainder, no more than 25% should be on-call, with up to another 25% on other operational work. The book derives a maximum of two incidents per 12-hour shift, based on roughly six hours to handle an incident end to end, including root-cause analysis, remediation and postmortem follow-up. It describes the median at zero pages per day, with a median above one per day as a warning sign.

Team size and rotation shape

The same chapter sets team-size minimums: at least eight engineers for a single-site rotation, so each person is on-call one week in eight, and at least six per site for a dual-site follow-the-sun rotation. Multi-site rotations are preferred where the load justifies them because they avoid night shifts. Most product teams cannot staff eight, and the honest answer is to merge rotations (a shared on-call across related services) or to reduce the page volume until a smaller group is sustainable. A rotation of three people on a noisy service is a resignation pipeline.

Rotation design choices and their trade-offs:

Design	Strength	Weakness	Fits when
Single-site weekly	Simple, deep context per week	Night pages hit one time zone	8 or more engineers, low night load
Follow-the-sun	No night shifts	Handoff overhead, timezone coordination	6 or more per site, global product
Primary plus secondary	Backup for missed pages and escalation	Doubles the on-call roster	Any user-facing service
Shared rotation across services	Larger pool, lower frequency	Responder knows fewer services	Small teams, strong runbooks

Pair every primary with a secondary and define the escalation timeout. The SRE book mentions typical response targets of 5 minutes for user-facing or time-critical services and 30 minutes for less sensitive ones; set your acknowledgement timeout accordingly and escalate automatically when it lapses.

Measure on-call as a load, then spend it

Treat on-call as a budget you track, in the same way you track the error budget. Useful measures: pages per shift, out-of-hours pages, pages per incident (the inverse of alert quality), percentage of pages with a runbook, and percentage of pages that required no action. Review them in a monthly operational review. When pages per shift exceeds your target, the correct response is a reliability or alert-hygiene sprint, not a request that people be more resilient. The SRE book recommends measurable goals for overloaded teams, for example fewer than two paging events per shift, and tuning noisy alerts toward a one-to-one alert-to-incident ratio.

Compensation matters, and it is a design input rather than an HR afterthought. Google’s book describes offering time off in lieu or cash for out-of-hours work, capped as a proportion of salary. Whatever the mechanism, uncompensated night paging produces attrition among exactly the senior people whose experience the rotation depends on.

Runbooks and automation

A runbook is a short, tested procedure linked from the alert that triggers it. The rule of thumb is that a runbook should let a competent engineer who has never seen this service take the first safe mitigation step within five minutes. Structure each one the same way: what this alert means in user terms, how to confirm it, the ordered mitigations from least to most invasive, how to verify recovery, who to escalate to, and the date it was last exercised.

Runbooks rot. Make staleness visible by recording the last-tested date and including a runbook review in every postmortem that touched one. Then look for the runbooks that are pure scripts: if a procedure is “run these five commands in order”, it should be automation with a human approval button, not a document. For recovery procedures that are rarely exercised but critical, such as restoring a cluster from backup, a tested runbook is the product; see our Velero Kubernetes backup and disaster recovery runbook for an example of the format.

Blameless Postmortems, Action Items and Error-Budget Policy

A blameless postmortem is a written record of an incident that focuses on contributing causes without blaming an individual or team. Google’s definition rests on an assumption that everyone involved had good intentions and acted on the information they had. The point is not kindness; it is data quality. If people expect punishment, they omit the awkward detail that explains the failure.

Figure 4: The learning loop. The error budget check is what turns a postmortem from a document into a scheduling constraint.

When to write one

Define triggers before incidents happen. The SRE book lists user-visible downtime or degradation beyond a threshold, any data loss, on-call intervention such as a rollback or traffic reroute, resolution time above a threshold, and a monitoring failure (usually an incident discovered manually). It adds that any stakeholder can request one. The monitoring-failure trigger is the most underused: an outage detected by a customer tweet is a detection-system defect even if the fix was trivial.

A postmortem template that earns its keep

Keep the template short enough that people complete it. A workable structure:

Summary in three sentences: what happened, user impact, how it was resolved.
Impact quantified: duration, affected users or requests, error-budget consumed, revenue or SLA exposure if known.
Timeline in UTC from the scribe’s log, including detection, declaration, each mitigation attempt and recovery.
Contributing factors, plural. Resist a single “root cause”; real failures are conjunctions of latent conditions and a trigger.
What went well, what went badly, where we got lucky. The “lucky” column is where near misses surface.
Detection and response review: did the alert fire on symptoms, how long until a human acknowledged, did the runbook help?
Action items, each with owner, due date, priority and a type (prevent, detect, mitigate, process).

Two review practices from the SRE book are worth copying. Senior engineers review drafts for completeness, impact assessment, depth of the contributing-factor analysis and priority of the action plan. And the finished document goes into a searchable repository with wide visibility, taking care never to include information that could identify end users. Reading clubs and a “postmortem of the month” keep the corpus alive; a repository nobody reads is a graveyard.

Action items: where most programs fail

The classic failure is a thorough postmortem followed by action items that rot in a backlog. Counter it mechanically. Every action item needs a named owner, not a team. It needs a due date within a quarter. It needs a type so you can see whether you are only ever adding detection and never removing causes. And it goes into the same tracker as feature work, because a separate “reliability backlog” is read by nobody with priority authority.

Report two numbers monthly: the percentage of postmortem action items closed by their due date, and the count of repeat incidents (same contributing factor as a prior postmortem). The second number is the real test of the learning loop. If it is not falling, the postmortems are theatre. Also avoid overloaded action lists: eight items of which two matter bury the two. Rank them, and label the top one or two as the ones that, if completed, would have prevented or sharply shortened this incident.

Language that preserves blamelessness

Blamelessness lives in phrasing. “The engineer should have checked” assigns fault; “the deploy tool did not surface the pending migration” describes a system. Ask “how did it make sense at the time?” rather than “why did you?” The goal is to understand the information, tooling and incentives that made the action reasonable. Where a person did make an error, the useful question is why the system allowed one error to reach production and why it took that long to notice.

Error-budget policy: the economic layer

The error budget ties incidents to decisions. The SRE Workbook’s example policy for a hypothetical game service shows the pattern. If the service exceeds its error budget over the preceding four-week window, halt all changes and releases other than P0 issues or security fixes until the service is back within SLO. The policy separates cases where reliability work is mandatory (a code bug or procedural error caused the overrun, or a postmortem shows a hard dependency could be softened) from cases where feature work may continue (a company-wide network problem, another team’s frozen service, or out-of-scope traffic such as load testers).

Two threshold rules in the example are directly relevant here. If a single incident consumes more than 20% of the error budget over four weeks, the team must run a postmortem with at least one P0 action item addressing the root cause. If one class of outage consumes more than 20% of the budget over a quarter, a P0 item goes into the next quarter’s planning. Disputes over budget calculation or required actions escalate to a named senior decision-maker (the CTO in the example).

The wording matters because it removes negotiation at the worst moment. Product and engineering agreed in advance what happens at exhaustion, so a freeze is the execution of a policy, not a political fight. Adopt the structure, then tune the numbers. A freeze is a blunt tool; many teams soften it to “all risky changes require a reliability review” for the first few cycles, and tighten later.

An honest caveat: an error budget only works if the SLO reflects user experience. If the SLI is a health-check endpoint that stays green during real failures, the budget never depletes and the policy never fires. Audit SLIs against past incidents: replay each incident and ask whether the SLI would have moved. If not, fix the SLI before trusting the budget.

Metrics Pitfalls: Why MTTR Misleads

Mean time to resolve (MTTR), along with mean time to detect (MTTD) and mean time to acknowledge, is the default scoreboard for incident programs. It is also a statistically weak one. The Verica Open Incident Database (VOID) analysis by Courtney Nash, summarised by InfoQ, found that incident durations are positively skewed, with most incidents resolving in under two hours and a long tail of outliers. For skewed data, a mean is dominated by a few extreme events, so a quarterly MTTR can move by a large amount because of one unusual outage, regardless of whether your process improved.

The same analysis reported no correlation between incident duration and severity, and concluded that duration does not convey anything useful about the event’s impact or the effort required to handle it. It also classed duration as “gray data”: high in variability, low in fidelity, since organisations define start and end times differently (when the fault began, when the alert fired, when someone noticed, when the status page cleared). The authors cautioned against swapping one simple metric for another and recommended looking at SLOs, customer feedback, sociotechnical incident data, post-incident review data and near misses.

What to measure instead

None of this means “stop measuring”. It means measure things with a clearer contract and report distributions, not means. Reasonable candidates:

Error budget consumed per incident and per month, which measures user impact directly and is comparable across incidents.
Time to mitigate, reported as median and 90th percentile, with the start-of-clock fixed to the first user-visible impact or the first alert, whichever is earlier, and written into your metric definitions.
Detection source mix: share of incidents found by alert versus by customers or by chance. A rising customer-reported share is a coverage alarm.
Pages per shift and alert-to-incident ratio, as on-call health indicators.
Repeat-incident rate and action-item closure rate, as learning-loop indicators.
Incident count by severity over time, read alongside change volume, to see whether delivery speed is outrunning reliability.

Present them as percentiles with a short narrative, not as a single trending number. And never use any incident metric as an individual performance target: the moment MTTR becomes a goal, people close incidents early and declare fewer of them, which corrupts the data and defeats the declare-early principle.

The Goodhart traps to watch

Targeting “fewer SEV-1s” produces downgraded severities. Targeting “faster acknowledgement” produces auto-acknowledge scripts. Targeting “more postmortems” produces templates filled with boilerplate. In each case, pair the quantitative signal with a periodic qualitative review (the SRE book suggests surveying teams on whether the postmortem process adds value or excessive toil) so that gaming surfaces as a conversation rather than going unnoticed.

Tooling Architecture and Build-Versus-Buy

Tooling should follow the architecture, and the architecture has a handful of integration points. The decision for each is whether to buy a product, assemble open source, or script it.

Component	Typical options	What to require
Telemetry and SLI store	Prometheus, OpenTelemetry Collector with a metrics backend	Consistent labels, enough retention for the longest alert window
Alert routing	Alertmanager, vendor alert rules	Grouping, inhibition, silences with expiry
Paging and schedules	PagerDuty, Opsgenie-class services, Grafana OnCall-class tools, in-house	Escalation policies, overrides, mobile push reliability
Incident coordination	Chat-ops bots, incident platforms	Auto channel creation, role assignment, timeline capture
Status page	Hosted or self-hosted	Independent hosting, so it survives your outage
Runbooks	Wiki, repo with markdown	Linked from alerts, owner, last-tested date
Postmortems and actions	Docs plus tracker, incident platforms	Templates, due-date tracking, searchable archive

Verify current product availability and licensing on each vendor’s site before committing; this market changes quickly, and some well-known tools have been retired or reorganised in recent years.

Independence of the incident tooling

One design principle deserves emphasis: the incident toolchain must not share fate with what it is managing. If your chat platform, status page and paging provider run on the same cloud region or identity provider as production, a major outage can blind you at the moment you most need coordination. Keep an out-of-band path: a secondary paging channel such as SMS or phone, a status page hosted separately, and a documented fallback bridge. PagerDuty’s severity guide even lists impaired notification pipelines and incident-response tooling as a SEV-2 condition, which reflects how seriously the failure of the tools themselves should be treated.

Where AI assistance fits

AI can help at three points without changing the architecture: summarising the timeline for handoffs, suggesting similar past incidents from the postmortem corpus, and drafting update messages for the Communications lead to approve. It should not hold the Ops role unattended in production. Our AIOps incident response architecture piece covers agentic triage and its guardrails. For AI systems themselves, which fail in unusual ways, see the telemetry conventions in OpenTelemetry for GenAI and LLM agent observability.

Trade-offs, Gotchas, and What Goes Wrong

Every element of this architecture has a failure mode. Naming them in advance is cheaper than discovering them mid-incident.

Alert fatigue disguised as diligence. Teams add alerts after each incident (“never again”) and never remove any. Within a year the paging volume exceeds what any rotation can absorb, and responders begin ignoring pages. Every postmortem should include an explicit “which alerts can we delete or downgrade” question alongside “which can we add”.

Severity inflation and deflation. If every incident is SEV-1, the label means nothing; if teams are judged on SEV counts, they deflate. Fix by tying definitions to measurable impact and by reviewing classification in postmortems, as PagerDuty recommends, rather than during the incident.

The hero IC. One senior engineer becomes the default commander for everything, which hides the fact that nobody else has practised the role. When that person is on holiday, the process collapses. Rotate the IC role deliberately, as the SRE book advises, and shadow new commanders.

Too many cooks. Open bridges attract observers who ask for status, interrupting responders. Provide a read-only stakeholder channel and a rule that questions go to the Communications lead. The same discipline stops well-meaning engineers from changing production outside the Ops lead’s coordination.

Postmortem without teeth. Documents are written, praised and ignored. The antidote is the budget policy and the action-item metrics above, plus leadership that visibly funds the top items. The SRE book explicitly recommends getting senior leadership involved and rewarding well-written postmortems.

Blameless as an excuse. Blamelessness is about not punishing individuals for system-induced errors; it does not mean no accountability. Repeated disregard of an agreed process, or concealing an incident, is a different matter and should be handled outside the postmortem.

Statistical traps in burn-rate alerting. Low-traffic services alert on a single failure; sparse labels hide per-tenant outages; recording rules that lag create false negatives; and a window longer than your metric retention silently returns nothing. Test each alert by replaying historic data or by injecting synthetic failures, which is another place chaos experiments pay off.

Cost. A well-run incident programme is not free. A paged IC plus an Ops lead plus a communications lead for two hours is real engineering time, and a secondary on-call doubles the roster. The offsetting argument is expected loss: even without precise numbers, compare the time invested in process against the cost of a longer, chaotic outage. Do the arithmetic with your own revenue and SLA figures rather than a generic industry statistic.

Practical Recommendations

Start small and close the loop before you expand the machinery. If you have to pick a sequence, begin with detection quality and ownership, then add command roles, then the learning loop, then the economic layer. Trying to introduce a full SEV-1 to SEV-5 programme before you can reliably page the right person is how process initiatives die.

A pragmatic first-quarter plan: choose your top three user journeys and write an SLI and SLO for each. Implement the fast-burn page and slow-burn ticket from this article for them, and delete at least as many cause-based pages as you add. Adopt a four-level severity scale with metric-based definitions. Write a one-page IC guide and run a Wheel of Misfortune session. Stand up the postmortem template and the action tracker. In the second quarter, add the error-budget policy with softened consequences, and begin reporting the on-call and learning-loop metrics.

Checklist:

[ ] Every paging alert fires on a symptom, links a runbook and has a named owning team.
[ ] Burn-rate alerts use paired long and short windows (for a 99.9% SLO: 14.4x over 1 hour and 5 minutes, 6x over 6 hours and 30 minutes, 1x over 3 days and 6 hours).
[ ] Severity definitions use user-impact percentages and map to a response, a communication duty and a postmortem rule.
[ ] Declaration criteria are written down; the default when unsure is to declare, and to take the higher severity.
[ ] A named IC exists for every declared incident; only the Ops lead changes production.
[ ] A scribe records a UTC timeline; handoffs are verbal and logged.
[ ] Rotations are sized for sustainable load, with a secondary and an automatic escalation timeout.
[ ] Pages per shift and alert-to-incident ratio are reviewed monthly.
[ ] Postmortem triggers are defined in advance; action items have an owner and a due date and live in the main tracker.
[ ] An error-budget policy states what happens at exhaustion and is signed by product and engineering leadership.
[ ] Incident tooling does not share fate with production.
[ ] Reports use percentiles and budget consumed, not a single MTTR.

Frequently Asked Questions

What is incident management in SRE?

Incident management is the coordinated process of detecting, triaging, mitigating, resolving and learning from events that threaten a service’s objectives. In SRE practice it includes defined severity levels, a command structure with an incident commander and separate operations and communications roles, runbooks, on-call rotations and written postmortems. Its purpose is to restore service quickly and, just as important, to convert each incident into prevention work that reduces the chance and impact of recurrence.

How many severity levels should we have?

Four or five is typical; PagerDuty’s public guide uses five, SEV-1 to SEV-5. What matters more than the count is that each level is defined by measurable user impact and bound to a response: who is paged, whether an incident commander is assigned, how often updates go out, and whether a postmortem is required. If responders regularly argue about which level applies, the definitions are too vague. When unsure, pick the higher level and reconcile afterwards.

What is a good burn-rate alert for a 99.9% SLO?

The Google SRE Workbook recommends paging on a 14.4x burn rate sustained over one hour and confirmed over five minutes, which consumes 2% of a 30-day budget. It also recommends paging on 6x over six hours with a 30-minute short window (5% of budget), and opening a ticket for 1x over three days with a six-hour short window (10% of budget). Requiring both windows reduces false pages and shortens alert tails.

How do you run a blameless postmortem?

Write it collaboratively soon after the incident, using the scribe’s timeline. Describe contributing factors in terms of systems, information and tools, never individuals. Record impact, what went well, what went badly and where you were lucky. List action items with owners and due dates, have a senior reviewer check the depth of analysis and priority of actions, and publish it in a searchable repository. Then track the action items in the same system as feature work.

How much on-call is sustainable?

Google’s SRE book suggests no more than 25% of an SRE’s time on-call, a maximum of about two incidents per 12-hour shift, and team minimums of eight engineers for a single-site rotation or six per site for dual-site. Treat these as starting anchors, not guarantees, because load depends on alert quality and system maturity. Track pages per shift and the share of pages needing no action, and respond to overload with alert and reliability work rather than asking people to endure it.

Is MTTR a good incident metric?

On its own, no. Research from the Verica Open Incident Database, discussed by Courtney Nash, found incident durations to be positively skewed and found no correlation between duration and severity, so the mean is dominated by a few outliers and says little about impact. Prefer the error budget consumed per incident, median and 90th-percentile time to mitigate with a fixed definition of the start of the clock, detection source mix, and repeat-incident rate.