Operating in Production: On-Call, Incident Command, and Reporting Clocks Module 6 · Own It

Saturday 02:14, Replayed

Last reviewed · content updated

Advanced

What you'll learn

~25 min
  • Replay one incident end to end - page, declare, command, clocks, comms, record, review - using only outputs already committed to the training's substrate
  • Read a complete NON-PRODUCTION incident record with determination entries and a corrective-action register that the training's validator accepts
  • Name what this training leaves for Training 09's autonomy switch and for the Resilience training's degraded state
ℹLeadership brief

What it is: the capstone replay — one incident, walked end to end through every capability this training built, using only outputs already committed to the record.

What it buys: proof that the seven-step system — page, declare, command, clocks, comms, record, review — produces one document a regulator, a contracting officer, and next quarter’s on-call can each read and find an answer in, before the first real Saturday at 02:14.

What to fund: nothing new. This lesson spends the capability the training’s six modules already funded, and names the two things it does not cover.

Before the detail — Artifact: the record itself, plus what this chapter hands the two trainings that follow it. Status of what follows: a worked replay of practice already established as binding or common baseline; nothing here is a new obligation.

Prompt first: run the checklist against your own incident

Here is our last real incident's timeline [paste: alerts, chat log,
tickets] and the incident-record template from Lesson 1.4 [paste].
Walk it through this training's seven steps and tell me, for each
one, whether we have an artifact or a gap:
1. PAGE - what fired, and was it actionable (2.1)?
2. DECLARE - what severity, by whom, into which state (1.2)?
3. COMMAND - who held the seat, and is there a timestamped
decision log with rationale (3.1, 3.2)?
4. CLOCKS - which reporting obligations applied, from which
trigger, and is each determination dated and attributed (4.1,
4.2)?
5. COMMS - what went out, on what cadence, to whom (4.3, 4.4)?
6. RECORD - does one document hold all of the above (1.4)?
7. REVIEW - was a postmortem trigger checked, and does every
corrective action have an owner, a due date, a verification
criterion, and closure evidence (5.1, 5.3)?
For every step with no artifact, write GAP - never an invented one.

The agent can hold a checklist against a real timeline; only the people who ran the incident can say whether a decision log actually exists, or whether item seven is being filled in for the first time under this prompt.

Page and declare

At 14:22 UTC the alert router paged: FieldDeskRequestsTooSlow had fired, the one-hour slow ratio at 28.5% and the five-minute ratio at 100% — 2.2’s burn-rate policy doing its job. t.nowak acknowledged, declared SEV2, and took the commander seat in two minutes: 1.2’s state machine moves from detected to declared at 14:24.

The seven minutes between the deploy and the page is the multi-window design working as intended, not a delay: the five-minute window crossed its threshold within three minutes, and the response waited for the one-hour window to agree before waking anyone — the trade that stops a pager firing on every blip.

A second signal arrived on a slower clock. Four days later, the cost-anomaly scan flags fielddesk-api’s database spend at $924.25 against a $375.63 baseline the day of the incident — a robust z-score of 43.3 — tailing three more days before it clears. Something scaled up during the outage and never scaled back, unwatched because the service had been healthy for days. That is 1.1’s third thread, resolved: the page answered in seven minutes, the runaway job in four days, on the cost signal 2.4 owns.

Two clocks ran on the same incident at two different speeds: both are real, and only one of them woke anybody.

Command, and the clocks that stopped

The record’s decision log carries three timestamped calls: at 14:24, declare SEV2 because crews had a radio fallback; at 14:33, roll back before root cause because exhaustion followed the deploy and no schema change blocked reversal — 3.2’s rule, mitigate first and let root cause wait; at 14:36, classify the event as a defect, not an intrusion, because no data left the system and no grid-control equipment was involved. Each row names what was known and what was assumed.

The last call records three negative regulatory determinations. The pending critical-infrastructure rule was both inapplicable and not in force; no grid-reliability duty applied because no control system was compromised; and the event was immaterial to investors because one workflow degraded for twenty-seven minutes with a working fallback and no customer or safety impact. Negative determinations let the record say nothing was reportable without saying nothing.

The scenario’s clock table also matched a six-hour DOE row because the scenario sets electric_utility: true, then records why system operations ruled it out: no customer lost service and no firm load was shed. Preserving that attributed reasoning avoids the audit finding created when an organization cannot show why it did not report.

Comms and the record

The internal-update template used at 14:31, when the status page told regional dispatch to fall back to radio, carries one field that turns a broadcast into a promise: a named next-update time, sent “even if nothing has changed.” Section 3’s timeline shows the promise kept: queued work orders synced and the status page cleared at 15:10, before the state moved to resolved at 15:12.

One of the timeline’s own rows is not a human’s entry. At 14:31, alongside s.duval’s status-page update, an assistant drafted the first reconstruction of the timeline, and the row validates only because it carries every field its own marker requires: a draft id, the evidence it read — lines 61 through 90 of the committed replay excerpt, plus 2.2’s committed rule-test transcript — a named verifier, j.whitfield, a verification instant five minutes later at 14:36, and what that verifier changed: the deploy timestamp and the first observed 504. A bare draft with no marker, or a marker missing one of those five fields, is exactly what the validator rejects; it cannot detect a fully unmarked draft, so an unmarked one still depends on a human noticing it.

Every piece above — the page, the declaration, the decision log, the determinations, the clocks, the comms, and now the one AI-drafted row — lives in one file, not seven. python3 incident/validate.py incident/incident-record.INC-2026-041.md runs against it and exits clean: incident artifacts ok (1 files, 0 warning(s)). This record’s section 4 was filled from the simulator-backed timeline, not produced by a tabletop replay — 6.1 showed the same record shape from a scripted exercise rather than the simulated FieldDesk scenario.

One command, run against one file, replaces an assessor’s sample with a repeatable check.

Review, and the corrective-action register

The review was held on 2026-08-21, three days after resolution. The record’s own postmortem trigger check names why: user-visible downtime, an on-call intervention (the rollback at 14:33), and a monitoring failure (/healthz stayed green through pool exhaustion) all read yes — the postmortem trigger fired three times over, before anyone had to argue that the night was bad enough to deserve a review. One row reads MISSING rather than no: nobody wrote down whether a stakeholder asked for a review on the bridge, and the check says so rather than guessing.

The corrective-action register gives every action an owner, a due date, a verification criterion, and closure evidence. One closes a saturation signal that never reached a dashboard, verified against a named board revision. Another makes the health probe exercise the real connection pool, verified by a timed staging drill. The connection-leak test remains open and states exactly what will close it: a link to the pull request that adds it.

That ledger, not a duration number, is what “the review’s product is change” looks like in practice: 5.3’s process measures — clock met, handoff clean, actions closed by their own due dates — read directly off this record rather than off a per-incident recovery-time trend nobody here is asked to defend.

Stop and escalate when a replay turns up a live gap — a clock with no owner, a determination never dated, a corrective action past due with no closure evidence. A gap found in a replay is a finding found for free; the same gap in a real incident is the finding a contracting officer writes instead. The risk acceptor who signed 1.3’s coverage attestation decides whether to fix it now or accept the exposure in writing.

KNOWLEDGE CHECK

This record carries three negative regulatory determinations - no grid-reliability report, no critical-infrastructure report, no investor-material disclosure. A reviewer suggests deleting section 5 next quarter, since 'nothing happened.' What should the record keep, and why?

Key takeaway

One incident, replayed end to end, is what this training built: a page that woke one person, a declaration into one state machine, a decision log that separates facts from assumptions, dated determinations that stopped clocks without staying silent, a promise kept on schedule, and a corrective-action register whose closed entries carry evidence and whose open entry names its closing condition. One file carries all of it, and a validator confirms that in one command.

Two things this training does not do, on purpose. It teaches an assistant that drafts under review and stops there — every draft carries the ai_drafted field naming who verified it, when, and what changed, as this record’s 14:31 row does — and the door to an agent that acts on its own, inside declared limits, with its own rollback and kill switch, is Training 09’s, the next training in this series. And it teaches the incident that resolves — mitigated, then resolved — never the service that stays degraded on purpose, running at a fraction of normal capacity because that is safer than an all-or-nothing failure; that is the Resilience training’s subject. Operating in Production ends here, with one document three different readers can each find their own answer in.

LEADERSHIP DECISION fund the incident as a designed system -
coverage, rotation, command, clocks,
communications, and review - before the first
real page, not after it
PRACTITIONER ACTION keep one incident record per event, timestamped
and validated, from declaration through the
corrective-action register's last closed row
SUCCESS MEASURE audit finding avoided: a regulator, a
contracting officer, and next quarter's on-call
can each read the same record and find their
own answer in it
Search lessons