Burn Rate Is the Page

Before the detail — Decision: page on how fast an availability budget is burning, proven by a unit test, not on a raw error count. Outcome: a policy that wakes someone within minutes of a real outage and falls silent within minutes of recovery, with the silence itself asserted. Artifact: the alert rule file and its test suite. Status of what follows: reusable guidance.

Prompt first: draft the policy, prove the silence

Here is our service-level-objective card (SLO - the availability
promise) [paste: service, objective, rolling window, and service-
level-indicator fields (SLI - the ratio measuring the promise)], and
the three-row burn-rate table we are told to implement: page at 14.4x
over 1h/5m, page at 6x over 6h/30m, ticket at 1x over 3d/6h.

Using ONLY the objective and metric names above:
1. Compute the absolute error-ratio threshold for each row
   (burn_rate x budget_fraction) and show the arithmetic.
2. Draft the three alert expressions, each window pair joined with AND,
   in our query language.
3. Draft one test fixture per row that asserts TWO things separately:
   the alert fires during a sustained breach, and it stops firing once
   the short window recovers even while the long window still
   remembers the outage.
4. Name, as a risk to check rather than assume absent, whether our
   metric could have an unlabelled total series sitting beside a
   labelled one on the same name.

Do not propose a target, a window, or an owner for the objective -
those come from the SLO card above. If the card is missing a field,
write MISSING and stop.

The objective has to stay an input the assistant never argues with — deriving it is a different lesson’s job entirely — and the fixture has to prove the silence as well as the fire, because a burn-rate alert with no test for when it goes quiet is a promise about relief nobody has checked.

The objective on screen is a given

FieldDesk’s committed SLO card is a given: 99.5 percent of eligible requests succeed over a rolling 30-day window (not calendar, so a team never gets a free reset on the first of the month), owned by r.okafor and approved by m.iqbal, reviewed quarterly. Its SLI excludes the /healthz health probe, which answered in milliseconds for the entire outage this training returns to because it never touches the connection pool real requests queued behind. Data Products 6.1 (a separate training in this series) covers where a number like that comes from: service levels derived from the decisions they support. This lesson takes 99.5 percent as given and builds only the page policy on top of it — a target the service cannot meet even while healthy, as 99.9 percent would have been here, is a broken promise, and catching that is a leadership call, not an alerting problem to route around.

The 30-day budget this objective buys is 216 minutes of fully-failed traffic — a number small enough that shipping a target nobody derived carefully is a contract risk hiding behind a decimal point, not a rounding choice.

Six ways to alert, and why five of them break

The Google SRE Workbook walks the same problem through five obvious approaches before landing on the one worth shipping, and each failure is worth knowing because each is someone’s first instinct:

1. Alert on raw error rate       - poor precision
2. Widen one window              - slow reset
3. Require persistence           - slow detection
4. Use one burn-rate window      - misses outage shapes
5. Add another burn-rate row     - reset still follows its window
6. Pair long and short windows   - sustained and still current

Row six is FieldDesk’s shape, and the only one of the six that pages fast on a deep outage and falls silent fast once it ends. The other five leave either a burn that slides past a single window, or a pager that keeps ringing long after the fix.

The rule, its test, and the point of the short window

The policy is three rows, not four — a widely repeated four-tier version does not match what the Workbook actually publishes. Two rows page, one files a ticket:

Every threshold in the rule file is that burn rate times the 0.5 percent budget fraction, computed once in the SLO file and restated in the rule as a comment rather than a bare number, so nobody has to reverse-engineer 7.2 percent back into “14.4 times the budget” during an incident.

Zero Trust 5.2 asks that a detection rule ship with the fixtures that exercise it, and 5.1 that someone witness it fire. This lesson owns the additional proof: exactly what a rule fires on and when it falls silent, asserted in a file that runs in seconds with no server.

Run the README command:

./bin/promtool test rules alerts/fielddesk_slo_test.yml

The committed transcript pins:

promtool, version 3.14.0 (branch: HEAD, revision: d7598b7141418fa35be2b5ec5d0fefb634199610)

$ promtool test rules alerts/fielddesk_slo_test.yml
  SUCCESS

Behind that one word sits the assertion that earns this section its title. At eighteen minutes into the scripted incident, the fast window already sees latency trouble but the slow window has not caught up — nobody is paged, correctly, because a blip that has not yet cost real budget should not wake anyone. At eighty minutes, thirty-five minutes after recovery, the one-hour error ratio remains 18.53 percent while the five-minute ratio is 0.06 percent; the fixture separately asserts the page is silent:

Without the short window this same alert keeps ringing at an engineer with nothing left to do for another forty minutes — the difference between a page that ends with the incident and one that trains a team to ignore its own pager.

Prometheus sits beside comparable tools in distinct license shapes worth knowing before any of them reach procurement:

Apache-2.0 and MIT are permissive licenses; AGPLv3 is a network-copyleft license. No price belongs on that table — billing units change faster than this lesson can track.

A three-row policy with no failing case ever asserted is a claim resting on hope; the same policy with a committed, hash-checked test suite is a control an assessor can re-run in minutes instead of reconstructing from a description.

A test that passes because it is wrong

The committed cause-rule fixture passes when FieldDeskConnectionPoolHigh fires for thirty minutes during ordinary traffic while errors remain 0.17 percent and symptom alerts remain silent; that passing test proves the cause alert is unsuitable for paging.

The mutation fixture isolates each branch: changing 14.4 to 999 makes the matching assertion fail. An unlabeled http_requests_total series matches the denominator but not the code="5xx" numerator, doubling the denominator and halving the computed burn rate. A [3d] selector is a lookback at evaluation time, not a three-day delay.

The scenario is deliberately tuned: an earlier ten-minute fault averaged about 2.3 percent over one hour and correctly did not page; the committed fifteen-minute fault produces the demonstration page. Optional linux-amd64 binaries are a 146-megabyte download; burnrate.py is only a no-download semantics check, not query-language validation. The committed transcript is hash-checked by the build.

Stop and escalate when a change to any burn-rate constant, window, or the label contract ships without a paired test proving both the new fire condition and the new silence condition — that is a change to what wakes a person at night, and it gets the review a rota change gets, from whoever owns the SLO, not a quick edit before a sprint closes.

Key takeaway

A given availability objective turns into a page policy through the multi-window, multi-burn-rate shape the Workbook’s progression arrives at after five weaker attempts fail: a long window proving the budget is really burning, a short window proving it still is right now. The unit test proves not just that a rule fires, but exactly what it fires on and when it correctly goes quiet — the t+80-minute assertion is the whole argument for the short window in one runnable line. A cause-based alert can pass its own test while paging on nothing a user feels, an unlabelled series can halve every threshold silently, and a bad window can turn “in three days” into “the moment this deploys.” Lesson 2.3 turns to who is actually on the other end of a page that survives all of this: the rota, and what a contract can require of it.

LEADERSHIP DECISION   require a paired fire-and-silence unit test for every
                      page-tier alert rule before it ships, the same review
                      bar as a production code change
PRACTITIONER ACTION   build the three-row policy on the given objective,
                      prove it with promtool, guard the thresholds with a
                      mutation test, and check for the unlabelled-series trap
                      before trusting any burn-rate number
SUCCESS MEASURE       zero page-tier rules in production without a committed,
                      hash-checked test proving both when they fire and when
                      they fall silent - an audit finding avoided per rule