Incident response
Incident response restores service first, fixes the cause second, and prevents the repeat third.
Outcome
| Part | This Process |
|---|---|
| Outcome | Each Incident is on the status page within five minutes, service is restored, and a postmortem links the Change that prevents a repeat. |
| Reads | Incident |
| Leaves | Incident, Change |
| On time | Within 4 hours (calendar time); the clock never stops. Late Actions go to the Engineering Lead. |
| Runs in | Each Startup |
Accountable
Reliability Engineer, Engineering Team.
Responsible: Reliability Engineer, Customer Lead.
Consulted: Engineering Lead.
Informed: Founder.
Workflows
| Workflow | Runs | Team |
|---|---|---|
| Resolve an incident | on Incident.created | Engineering Team |
Who does each step
| Workflow | Step | Role | Calls | Approval |
|---|---|---|---|---|
| Resolve an incident | 1. Acknowledge the Incident and post it to the status page within 5 minutes. | Reliability Engineer | Incident.acknowledge | None |
| Resolve an incident | 2. Restore service, reverting the last Change when it is the cause. | Reliability Engineer | Change.revert | None |
| Resolve an incident | 3. Fix the cause in a new Change. | Reliability Engineer | Change.create | None |
| Resolve an incident | 4. Write the postmortem and link the Change that prevents a repeat. | Reliability Engineer | Document.create | None |
| Resolve an incident | 5. Resolve the Incident and update the status page. | Reliability Engineer | Incident.resolve | None |
Events
| Direction | When | What happens |
|---|---|---|
| In | On Incident.created | Starts Resolve an incident |
| Out | Incident.acknowledged | Raised by step 1 of Resolve an incident |
| Out | Change.reverted | Raised by step 2 of Resolve an incident |
| Out | Change.created | Raised by step 3 of Resolve an incident |
| Out | Document.created | Raised by step 4 of Resolve an incident |
| Out | Incident.resolved | Raised by step 5 of Resolve an incident |
Plays
No Play names this Process.
Controls
- Deploy reconciliation
- Anomaly monitoring
- Incident handling
- Revert by query
- Breach notification
- Capacity headroom
- Point-in-time recovery
Settings
| Setting | Type | Default | Say | Set with |
|---|---|---|---|---|
incidents.statusPostWithin | string | 5m | Post every incident to the status page within five minutes | Setting.update |
Failure
A failed step retries; when retries run out, the Action goes to the Engineering Lead.
| Step | On failure |
|---|---|
| Resolve an incident 1. Acknowledge the Incident and post it to the status page within 5 minutes. | Times out after 30 minutes. Goes to the Engineering Lead. |
Loop
No loop: the path is never randomized, because it serves Incident handling.
Code
Turn on incident responseEnable
On by default for every Product Template. Process.pause stops new Actions; Actions in progress finish.
Post every incident to the status page within five minutesExample
Lesson booking fails after a deploy. The Incident is on the status page in 3 minutes and the Change reverts in 5. The postmortem names the missing test, and the fixing Change adds it.