Rehearse the Response
Last reviewed · content updated
AdvancedWhat you'll learn
~18 min- Distinguish an incident exercise from a restore drill, and rehearse declaration, command, handoff, evidence preservation, and communications together
- Run a tabletop exercise whose emitted decisions produce a complete incident record that the training's validator accepts unchanged
- Meet a federal exercise cadence with a dated, named record as the evidence an auditor checks for
What it is: an incident exercise — a rehearsal of declaration, command, handoff, evidence preservation, and communications, run against a scripted scenario instead of a real page.
What it buys: proof, before the first real Saturday at 02:14, that the coverage decision, the rota, and the clock-owner role actually work under pressure — and a dated record when an auditor asks whether the plan was tested, not just written.
What to fund: recurring drill time on a calendar, a written exercise record for every cycle, and the standing commander pool and clock-owner roles the drills exist to rehearse.
Before the detail — Artifact: the exercise record — the same shape as an incident record, produced by a drill instead of an incident. Status of what follows: binding cadence wherever a named federal standard reaches Meridian’s business; common baseline everywhere else.
Prompt first: draft the injects, not the answers
Here is our service's architecture summary [paste], our on-callrota [paste Lesson 2.3's shape: tiers, backups, escalation], and ourseverity matrix [paste Lesson 1.2's shape].
Draft THREE tabletop injects for a game-day exercise. For each one: - a time and a one-line trigger a responder would actually receive - two or three plausible choices, each naming the ASSUMPTION it commits the responder to - never mark one "correct" - one sentence a facilitator would say afterward about the trade-off between the choices
Pick failure points from OUR architecture and OUR contract, not ageneric list. Where our runbook or contract has no named answer forwho declares, who commands, or who owns a clock, write MISSINGinstead of inventing a role to fill the gap.
Do not have the assistant pick a winning path or fix the service -only the people running the exercise read the transcript afterward,and grading the choice is not the point of the drill.The agent’s value is producing an inject that reads like a real page arriving at 02:10, not a training slide; only the people who will actually run the incident can supply the discomfort of choosing under incomplete information, so the prompt never asks it to grade the answer or supply a missing role itself.
Exercises are not restore drills
Cloud Modernization 4.3 (a separate training in this series) owns restore drills; this lesson owns incident-response exercises. Federal Delivery 4.4 contributes the rehearsed clock; here it is exercised with declaration, command, handoff, evidence preservation, communications, and clocks. Module 19 lesson 9’s five-phase shape — discovery, configuration, automation, verification, troubleshooting — remains the runbook form; this lesson rehearses the incident roles and clocks around it instead of re-teaching it.
Google’s Wheel of Misfortune trains that readiness directly: pull a name, hand them a scripted failure, and see if they can run it — “a validation that you could have solved this week’s problem if you had been picked.” The SRE Report 2026 (Catchpoint, n=418, published 2026-01-22) found that 34% of respondents said their organization has never tested a failure in production at all. That result shows production-failure testing is often skipped; it does not establish whether respondents rehearse coverage, command, or clocks in non-production exercises.
An exercise that never happens is not a savings. It is calendar time spent finding out whether the coverage attestation holds during a real Saturday instead of a Tuesday afternoon with a facilitator in the room.
The drill where the commander has no terminal
Command Is a Role (3.1) sets the rule: the commander decides and does not fix, and operations is the only group that touches the running system. A drill rehearses that rule by removing the option to break it — take the terminal away from whoever holds the commander seat, and command becomes visibly a role instead of a habit. A commander with no terminal can only ask questions, assign work, and decide; if they cannot resist reaching for a keyboard, the exercise found the gap before a real incident did.
The same drill can exercise the handoff (3.3): swap commanders mid-exercise, mid-inject, and require the spoken acknowledgment before the outgoing commander is allowed to leave. A handoff that only ever happens on a call nobody is testing is a handoff nobody has actually rehearsed.
This drill returns a fact leadership can act on: whether the pool of people who can hold the commander seat is one person deep or several — a coverage-decision detail (1.3) found during rehearsal rather than during the first real incident.
The tabletop that writes the record
A drill that produces only a transcript produced a game: interesting to the room, invisible to everyone else. The substrate’s tabletop tool is built to fail that standard on purpose. Run from the repository’s substrate directory:
python3 tabletop/tabletop.py --auto --emit-decisions /tmp/incident-record.INC-2026-052.mdpython3 incident/validate.py /tmp/incident-record.INC-2026-052.mdThe first command plays a scripted exercise — a nightly data export exposed for six days, discovered by a customer email rather than an alert — and writes a complete incident record, with section 4, Decisions under uncertainty, filled from the choices made rather than left blank. Every choice in the exercise names the assumption it commits the chooser to, and the tool says why in its own notes: “a decision row with an empty ‘what was assumed’ column is the one a reviewer cannot use.” The second command runs the same validator a real incident record has to pass, against the file the exercise just wrote, and it exits clean: incident artifacts ok (1 files, 0 warning(s)).
Five minutes after declaration, a second inject hands off command: the commander who took the seat at first contact has a flight to catch and must leave. 3.3’s rule applies immediately — only a spoken read-back of every open item, confirmed before the outgoing commander leaves, closes it. The emitted record carries that read-back as its own sub-table, separate from the decision log — an outgoing name, an incoming name, an acknowledgment timestamp, and the open items handed over, no blank cell allowed.
That pairing is the whole design. Decide Before You Know (3.2) teaches the timestamped decision log with the commander’s rationale; a tabletop that cannot produce one has rehearsed nothing this training can use as evidence. Its section 8, the corrective-action register, is the identical four-field register 5.3 already requires — owner, due date, verification criterion, closure evidence — used here for a drill’s trigger instead of a real one.
Stop and escalate when an exercise surfaces a gap that would leave a required federal clock unowned, a coverage tier unstaffed, or a determination with no named decider — a MISSING found in a drill is real in production. The exercise record’s own corrective-action register names an owner and a due date for the fix, and 1.3’s risk acceptor decides whether operating until it closes is coverage the organization can accept in writing or a rota that has to be fixed first.
A team runs a well-attended, blameless 'restore day' every quarter: they restore last night's backup to a scratch environment and confirm it boots. Their VP wants to report this as evidence of tested incident readiness. What is missing?
The commercial starting practice is a voluntary game day, run on a team’s own schedule and skipped first when a sprint deadline gets tight — Google’s Wheel of Misfortune is exactly that. The federal delta is a dated test cycle that two different kinds of regime name outright and do not treat as optional: a grid reliability standard requires the incident-response plan tested on a fixed calendar cycle with a dated lessons-learned report as the evidence an auditor checks for, and a federal breach-response policy requires an annual tabletop for the agencies whose flow-down reaches Meridian — flow-down being the clause that carries a federal obligation to a company that never signed with the government. The handoff artifact is the exercise record itself — timestamped, filed where an auditor can find it, in the same shape as a real incident record. What is not equivalent is that a voluntary game day proves something to the room that ran it; a dated test cycle is itself the evidence a regulator inspects, whether or not the drill turned up anything wrong.
Practice status — among organizations that run production incidents, commercial and federal
| Practice | Status | Also called |
|---|---|---|
| game-day rehearsal of command and clocks | common baseline | incident simulation / fire drill |
| drill where the commander holds no terminal | emerging | none in common tooling - this training’s drill |
| dated incident-response-plan test, 15 calendar months (NERC CIP-008 R2.1 - the grid reliability standard’s IR-plan test) | required where CIP-008 covers the system; strong optional elsewhere | incident-response plan test, with a lessons-learned report as named evidence |
| low-impact plan test, 36 months (CIP-003, the low-impact counterpart standard) | required where CIP-003’s low-impact category applies; not applicable elsewhere | none - a scaled-down federal requirement |
| annual tabletop (M-17-12 Section X - OMB’s 2017 breach-response policy for agencies) | required for the covered agency Meridian’s notification feeds; reference-shop for Meridian itself absent a flow-down clause | annual IR tabletop |
| exercise tool that writes a validator-accepted record | emerging | none in common tooling - this training’s —emit-decisions |
Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging
Key takeaway
An incident exercise and a restore drill test different things: one proves data comes back, the other proves the people and the clocks work under a scripted failure. Google’s Wheel of Misfortune calibrates individuals; the drill where the commander loses the terminal calibrates the role itself; and the tabletop’s emitted decisions turn a rehearsal into an artifact a validator — and eventually an auditor — will accept, using the same corrective-action register 5.3 already requires. A grid reliability standard and a federal breach-response policy each name their own cadence for the same rehearsal, with a dated record as the evidence they check for. Lesson 6.2 turns from what the exercises cost in calendar time to what running all of it — coverage, rotation, exercises, and communications — costs in dollars and hours.
LEADERSHIP DECISION fund a recurring incident exercise, distinct from the restore drill, with its own dated record on filePRACTITIONER ACTION run the drill with the commander's terminal removed, and produce a record the emit tool - or its equivalent - writes and the validator accepts, every cycleSUCCESS MEASURE audit finding avoided - a dated exercise record already on file before the next test cycle or annual tabletop is ever asked for