Policy as Code You Can Test

Before the detail — Decision: test the envelope’s decisions the way any other code that gates a real effect gets tested — fail-closed, on purpose, against cases that try to break it. Outcome: a policy file two independent evaluators decide identically on every case they are asked, plus a documented trap neither evaluator hides. Artifact: a 22-of-22 agreement table with its surface stated exactly, and a broken-on-purpose copy of the policy that proves why the fix matters. Status of what follows: reusable guidance; this lesson does not re-teach policy-as-code generally — DevSecOps 3.x (a separate training in this series) owns that ground — it owns testing one envelope’s decisions for fail-closed behavior. Demonstrated with NON-PRODUCTION classroom artifacts.

Prompt first: ask what a missing case decides

Read-only: for this envelope's policy file, list every action, every
zone, and every declared parameter bound. For each one, ask: if a
request arrives for an action, zone, or parameter this file has never
heard of, what does the policy decide - allow, deny, or nothing at
all?

Where the answer is "nothing at all," or you cannot tell without
running it, flag that line. A policy that goes silent on an unknown
case is not the same as a policy that denies one.

“Nothing at all” is the answer this lesson spends most of its time on, because a policy engine can produce it without ever raising an error — which means nobody notices until the case that exposes it actually happens.

Fail-closed, and tested for it

The envelope’s own default is deny, with no wildcard: an action absent from the file is refused, not ignored. That default only means something if the tests that exercise it actually try the cases that would break it — an over-bound parameter, an action nobody declared, a standing deny that must be receipted by name rather than as a generic refusal, and a state path pointed somewhere the agent has no business writing. Each of those is a case in this training’s own test suite, run against a temporary copy of the policy rather than the live one, so a test that fails cannot leave a stray file behind.

The subtler case is a missing input rather than a wrong one. The policy cannot compute how many actions a run has already taken — that count arrives as prior_actions, supplied by whoever holds the receipt log — and a policy that treated a missing count as zero would hand every first request of a freshly reset counter a free pass. The shipped rule instead treats an absent state_path, prior_actions, or prior_mutating as a violation, not a permission: a missing input is not the same fact as an input that says no. Fail-closed is not a slogan here; it is a specific answer to what a policy does when it does not know something.

The negative tier mutation-tests seven invariants one at a time. For the crew-ceiling case, it changes pre_position_crew.crews.max from 3 to 14 and requires mu_invariants.py to exit 1; request-only tests would not prove that an unauthorized widening stops the build.

One policy, two engines, and exactly what they were asked

policy/envelope.json is read by two independent evaluators — a small stdlib Python evaluator built for this training, and OPA (Open Policy Agent), a general-purpose policy engine reading the identical file through Rego, the policy language OPA runs. policy/agree.py replays 22 requests through both and requires them to agree on every one. Five of the 22, for a sense of the shape:

action                      params                                           expect  stdlib  opa     context
draft_work_order            {"crews": 4, "feeder_id": "F-2291"}              deny    deny    deny
pre_position_crew           {"crews": 14, "feeder_id": "F-2291"}             deny    deny    deny
send_dispatch_notice        {"audience": "all_crews", "feeder_id": "F-1105"} deny    deny    deny
delete_backup               {"backup_id": "bk-2026-08-18"}                   deny    deny    deny
draft_work_order            {"crews": 3, "feeder_id": "F-2291"}              deny    deny    deny    prior_mutating=4

22 of 22 cases agree (stdlib evaluator vs OPA, one policy file)

The agreement table’s surface is stated exactly, and it is worth reading for what it excludes as much as what it covers: it is the tool — present, absent, or declared deny — every declared parameter and its type, minimum, maximum, enum, and pattern, the target host, the action’s network zone, the state path a run is pointed at, both run budgets, and the approval count test — enough unexpired tokens bound to this request, from enough distinct approvers holding enough distinct keys. Deliberately not on that surface: the HMAC (a keyed signature only a shared secret can verify) on a token, the single-use consumption of a request id, the chain counting that produces the budget inputs in the first place, and everything after the decision itself — execute, verify, compensate, receipt. Those four are machinery, not policy, and a table that claimed to test them would be claiming more than it does. The zone rule is on the evaluator surface but has no row among the 22 cases; the shipped envelope contains no OT action, so the Rego tests, invariant, and negative test exercise that denial separately.

OPA is optional here — a 58.8 MiB binary that the fetch script downloads and checksum-verifies — and the stdlib evaluator remains the classroom default. The committed OPA transcript reports PASS: 25/25; together with the 22-case agreement table, that supports evidence for the tested behavior rather than an unsupported claim that “the policy is correct.”

The rule that isn’t false — it disappears

Rego’s most dangerous trap is not a rule that evaluates wrong; it is a rule that does not evaluate at all. Written the obvious way, rule := data.envelope.tools[input.action] is undefined for an action nobody declared — and in Rego, undefined is not the same value as false. Every later rule that depends on it becomes undefined too, and the entire decision document disappears rather than returning a deny. A gateway that reads “no decision” as “no objection” has just turned every tool it has never heard of into an allowed one, silently, with no error to catch.

The shipped policy avoids it with object.get(data.envelope.tools, input.action, {}), which is total: an unknown action produces an empty rule the violation checks can still reason about, and the same input against a deliberately broken copy of the file with the naive line restored proves the difference side by side — the shipped policy returns a readable deny; the broken copy returns no value at all, on the same request. The same family of bug bites a builtin called on a missing field: not is_string(input.state_path) does not fire when state_path is simply absent, because a builtin given an undefined argument is itself undefined — which is why the shipped rule writes object.get(input, "state_path", null) instead. Neither fix is exotic; both replace an assumption that every case will be asked with a default for the case that will not be.

The bypass test is part of the policy, not separate from it

A suite of unit tests over the decision function proves the function is correct. It does not by itself prove the function is the only way in — that is Lesson 2.1’s raw-request test, run against the real path rather than a mocked one: the same curl that reaches the target directly, once before the endpoint required a signed request and once after. A policy-as-code suite that never asks whether a call can skip the policy entirely has tested the logic and left the path untested, and the path is where an unenforced boundary actually costs something.

The ordinary gate hash-verifies the committed BEFORE/AFTER transcripts; T9_GATE_FULL=1 reruns and diffs both live fixtures. A production gate should rerun its equivalent whenever an enforced path changes.

Stop and escalate when a policy change adds a new action, zone, or parameter and the agreement table or the bypass test has not been re-run against it. A rule that type-checks but was never asked the question a real request would ask is untested, whatever its passing suite claims — and deciding whether the change ships anyway is a call for whoever owns the gate, not for whoever wrote the one-line diff.

Key takeaway

An envelope’s default-deny is only as real as the tests that try to break it: an over-bound parameter, an undeclared action, a missing input treated as a violation rather than a free pass. Two independent engines agree on 22 stated cases and OPA’s own suite passes 25 more, with the agreement table’s surface — and its four deliberate exclusions — stated plainly rather than assumed. The most dangerous failure mode is not a wrong answer but no answer at all, which is why the undefined-rule trap and the raw-request bypass test both belong in the same suite as the ordinary unit tests. Lesson 2.4 asks what still holds when every test above passes and the target still gets the wrong call anyway — the floor underneath all of it.

LEADERSHIP DECISION   require a policy change to pass the agreement
                      table and the bypass test before it ships, not
                      only its own unit tests
PRACTITIONER ACTION   write every policy rule as total - a default
                      for the case nobody declared - and re-run the
                      bypass test on every change to the file
SUCCESS MEASURE       zero policy changes shipped without a passing
                      agreement-table run against the new case - an
                      audit finding avoided