Skip to content

Continuity

Continuity sets recovery objectives per Capability, fails over between upstreams, names a successor signer, and runs restore drills on a cadence.

Outcome

PartThis Process
OutcomeEach Capability meets its recovery objectives in a passing restore drill, and a successor signer is on record.
ReadsRecovery Objective, Capability
LeavesRecovery Objective, Issue
Runs inEach Startup

Accountable

Reliability Engineer, Engineering Team.

Responsible: Reliability Engineer.

Consulted: Engineering Lead, Risk Analyst.

Informed: Founder.

Workflows

WorkflowRunsTeam
Drill a restoreevery quarterEngineering Team

Who does each step

Continuity: who does each step5 steps in 1 Workflow, across 1 Role: Reliability Engineer. The table below lists every step.Reliability EngineerOttoWorkflowDrill a restore1 · RecoveryObjective.listList each Capability whose restore drill isdue.2 · RecoveryObjective.drillRestore each Capability’s records into anisolated namespace and time it.3 · Integration.syncFail over to each fallback upstream in test,then back.4 · Role.getConfirm the successor signer is named andreachable.5 · Issue.createOpen an Issue for every objective the drillmissed.Continuity: who does each step5 steps in 1 Workflow, across 1 Role: Reliability Engineer. The table below lists every step.WorkflowDrill a restoreReliability Engineer1 · RecoveryObjective.listList each Capability whose restore drill is due.Reliability Engineer2 · RecoveryObjective.drillRestore each Capability’s records into an isolatednamespace and time it.Reliability Engineer3 · Integration.syncFail over to each fallback upstream in test, thenback.Reliability Engineer4 · Role.getConfirm the successor signer is named andreachable.Reliability Engineer5 · Issue.createOpen an Issue for every objective the drill missed.
Each box is a step, numbered within its Workflow, with the Verb it calls, under the Role it runs as. The table below adds approvals.
WorkflowStepRoleCallsApproval
Drill a restore1. List each Capability whose restore drill is due.Reliability EngineerRecoveryObjective.listNone
Drill a restore2. Restore each Capability’s records into an isolated namespace and time it.Reliability EngineerRecoveryObjective.drillNone
Drill a restore3. Fail over to each fallback upstream in test, then back.Reliability EngineerIntegration.syncNone
Drill a restore4. Confirm the successor signer is named and reachable.Reliability EngineerRole.getNone
Drill a restore5. Open an Issue for every objective the drill missed.Reliability EngineerIssue.createNone

Events

DirectionWhenWhat happens
InEvery quarterStarts Drill a restore
OutRecoveryObjective.drilledRaised by step 2 of Drill a restore
OutIntegration.syncedRaised by step 3 of Drill a restore
OutIssue.createdRaised by step 5 of Drill a restore

Plays

No Play names this Process.

Controls

No Control names this Process or a Workflow of it.

Settings

SettingTypeDefaultSaySet with
continuity.drillEverystringquarterlyDrill a restore every quarterSetting.update

Failure

A failed step retries; when retries run out, the Action goes to the Reliability Engineer.

Loop

StepValue
HypothesisQuarterly restore drills keep every restore within its recovery objectives.
MeasureRestore drills passed over 1y, at least 4 observations
LearnThe learn step reads the Readout, records Evidence and a Decision, and drafts the next Variant.

Code

Text
Turn on continuity

Enable

On by default for every Product Template. Process.pause stops new Actions; Actions in progress finish.

Text
Drill a restore every quarter

Example

Each quarter the Reliability Engineer restores the payments records into an isolated namespace. A restore that misses its 1h objective opens an Incident, and service fails over to a second upstream.