Incident Communication

Option B — a structured P1 opener with all required components.

P1 initial alert message structure:

[SEVERITY] INCIDENT — [TIMESTAMP UTC]
[WHAT IS BROKEN]: [specific service/endpoint]
[OBSERVED BEHAVIOR]: [exact error, status code, behavior]
Affected: [who is affected + scale]
Impact: [revenue, data, business consequence]
IC: [Incident Commander name/handle]
Bridge: [channel link or call link]
ETA for next update: [time]

Each component explained:
• Timestamp (UTC): establishes the incident timeline; always use UTC for global teams
• What is broken: specific: "Payment service / EU checkout" not "the site"
• Observed behavior: exact: "HTTP 500 for all EU checkout requests" — prevents teams from investigating different symptoms
• Affected users + scale: "~2,000 users/min" — quantified; enables triage prioritisation
• Impact: "Revenue impact: ongoing" — business consequence, not technical description
• IC: who owns the incident; single point of coordination
• Bridge: where to convene — prevents fractured communication across multiple channels
• ETA for update: sets expectations; prevents the channel from flooding with "any update?" messages

Option B — a timestamped update with progress report, current hypothesis, and a firm next update time.

Incident status update structure:

[TIMESTAMP UTC] UPDATE — [Service] [Severity]
[Investigation progress]: [what has been ruled out / what is being pursued]
[Current hypothesis]: [specific — names the component + mechanism if known]
[Next update]: [specific time — not "soon"]

Key elements:
• Timestamp on every message: the incident channel is a log; timestamps enable the postmortem timeline
• "UPDATE" prefix: clearly distinguishes updates from discussion
• Ruling out: "ruling out the 08:55 deployment" — shows progress even before finding the cause
• Current hypothesis: "connection pool exhaustion in EU-WEST-1 gateway" — gives the team a focus; others can validate or challenge
• Next update time: always a specific time, never "soon"; typically every 15–30 minutes for active P1

Why regular cadence updates matter:
• Stakeholders stop pinging ICs when updates are on a schedule
• Legal and finance teams need timestamps for impact reporting
• The postmortem timeline is built from these updates
• Even "no progress" updates signal that the incident is being actively managed

Option C — a complete all-clear with resolved time, root cause, fix description, duration, impact numbers, and a postmortem link.

All-clear message structure:

[TIMESTAMP UTC] RESOLVED — [Service] [Severity]
Services restored at: [timestamp UTC]
Root cause: [specific mechanism — what was wrong and why]
Fix: [what was changed + deployment time]
Duration: [total incident duration]
Affected [users/transactions]: [quantified]
Postmortem: [link/channel] by [deadline]

Critical components:
• "RESOLVED" prefix: explicit signal that the incident is closed
• Root cause in one sentence: "connection pool max size was 5 in EU-WEST-1 config (misconfigured during last week's parameter tuning)" — specific, causal
• Fix in one sentence: "max pool size increased to 50; config redeployed at 10:44"
• Quantified impact: "~5,800 affected transactions" — needed for customer SLA review
• Postmortem deadline: creates accountability; "by EOD Friday" not "eventually"

Why "Fixed, should be working now" is insufficient:
• No root cause means the next engineer doesn't know what to look for
• No impact number means no SLA reporting
• No postmortem link means lessons are lost
• "Should be working" expresses uncertainty; after confirmation, use "is working"

Root cause — the underlying systemic condition that enabled the incident to occur. It is never a person, and almost never a single event.

Structure of a postmortem root cause statement:

[The systemic condition] [that allowed/enabled] [the proximate cause] [to cause the incident] [without being caught by safeguards].

Good root cause examples:
• "The connection pool limit was not validated in CI/CD, allowing a misconfigured value to reach production." → Points to the missing validation, not the person who set the value
• "The alerting threshold for memory usage was set above the container limit, so the alert never fired before OOM." → Points to the monitoring gap
• "There was no canary deployment process for config changes, so the misconfiguration was applied to all instances simultaneously." → Points to the deployment gap

Common root cause anti-patterns:
• "Human error" — not a root cause; someone can always make an error; the question is why safeguards didn't catch it
• "Developer X changed the config" — this is the proximate trigger, not the systemic cause
• "The HTTP 500 error" — this is the symptom, not the cause
• "The Tuesday deployment" — this is the event; the root cause is why the deployment could reach production in a broken state

Good postmortems always ask: "What process, check, or automation was absent that allowed this to happen?"

Root cause — the primary necessary condition. Contributing factors — conditions that amplified the severity or duration without being independently sufficient.

Example distinction:

Root cause:
The connection pool limit was not validated in the infra config deployment pipeline, 
allowing a misconfigured value (max: 5) to reach production.

Contributing factors:
1. Monitoring alert threshold was set to fire at >80% pool utilization, 
   but traffic hit 100% in under 2 minutes — alert fired too late.
2. The EU-WEST-1 pod auto-scaling was disabled during last week's 
   maintenance window and not re-enabled.
3. The on-call runbook for connection pool errors had not been updated 
   since the gateway migration in Q3, delaying diagnosis by ~20 minutes.

Why the distinction matters:
• Root cause gets the primary remediation action (add config validation to pipeline)
• Contributing factors each get their own separate action items in the postmortem
• Without distinguishing them, teams may focus remediation on the wrong problem

Postmortem vocabulary:
• Root cause: "the incident was caused by…" "the primary failure was…"
• Contributing factors: "which was compounded by…" "this was made worse by…" "delayed detection because…"
• Detection gap: why it took N minutes to notice
• Resolution gap: why it took N minutes to fix after detection

Option B — a complete, specific, measurable, time-bound action item with an owner and a ticket.

Postmortem action item requirements:
• Specific: "Add a PagerDuty alert for connection pool utilization > 60% on all EU gateway pods" — not "fix monitoring"
• Owner: @devops — a named person or team; "team" is not an owner
• Due date: "2024-03-22" — specific; "ASAP" or "soon" are not acceptable
• Ticket: #4521 — tracked in the backlog; without this, action items disappear

Action item anti-patterns:
• "Fix the monitoring" — what exactly? which metric? what threshold? what alert channel?
• "Improve the deployment process" — too broad; split into specific, testable actions
• "Make sure this doesn't happen again" — not actionable; cannot be verified as done
• No owner → no accountability; assigned to "the team" → no one owns it

SMART action item template:

- [Specific action verb]: [what exactly will be done]
  - Addresses: [which contributing factor or root cause gap]
  - Owner: [@name or team]
  - Due: [specific date]
  - Ticket: [#id]

Postmortems that produce vague action items are postmortems that produce repeat incidents. The test: can someone verify this action item is complete in 30 days without asking for clarification?