Applied AI-Accelerated Government Development

Agents on the Parity Bench

Updated

Intermediate

What you'll learn

~55 min
  • Measure how strong the golden corpus is with mutation testing (deliberately breaking the code to see whether any test notices), and find the rule branch no case reaches
  • Have an agent propose new inputs while only the legacy system supplies expected outputs, and replay a recorded property run (generated inputs with recorded legacy answers) in every build
  • Put the referee files (tolerances, golden outputs, rules, the harness and its tests) out of the agent's reach with code owners (named teams whose review a change to their files requires), and check that the ownership is declared
  • Record every mismatch as a ruling made by a named person on that mismatch's exact evidence, and block cutover until each one is ruled

Lesson 23’s harness passes when the new feature matches ten recorded cases. That proves the rules those ten cases exercise, and nothing else. An agent is very good at the work that widens that proof: reading rules.md, finding the inputs nobody tried, running thousands of them, and sorting what differs. It is also the participant with the strongest incentive to make a red run green. This lesson gives the agent the first job and takes the second one away.

Three rules hold the whole design together. The agent proposes inputs; only the legacy system supplies the answer for an input. The agent never grades its own work; a deterministic harness does. And the agent never edits the referee, meaning the files that decide pass or fail.

Before the detail — Artifact: a mutation-testing step, a recorded property set with a replay test, a list of referee paths owned by code owners with a check of that ownership, and mismatch proposals and rulings with a checker. Acceptance test: with one branch of the rounding rule deleted, the ten golden cases still pass and the replay fails on 13 of 200 recorded inputs; a mismatch with only the agent’s proposal on file fails the ruling check. Status: reusable guidance; mutation and property testing are published practice, the referee and ruling files are this module’s.

⚠The bench runs on restores, and restores are case data

The bench below runs both systems against restores of the real database, which keep the source data’s handling rules: for a case database, CUI (controlled unclassified information; in a commercial shop, think customer PII under contract). The agent in this lesson never needs a case row: it reads rules.md, mutation results and the observations of synthetic inputs it proposed. Give it no database credentials. The bench runner, a pipeline job with its own identity, does the reading. Where the CUI determination from lesson 01 does not cover even that, build the bench from a masked or synthetic snapshot instead.

Who does what on the bench

ParticipantMayMay not
the agentread rules, unknowns and mutation results; propose candidate inputs; propose a classification for a mismatch; fix the new codewrite an expected output, edit a referee file, rule on a mismatch, hold database credentials
the bench runner (a pipeline job)refresh restores, post inputs to both systems, read the rows they wrote, write the parity reportchange the harness it runs
the legacy copyanswer—
a named personrule on a mismatch, promote a recorded candidate into the golden corpus, change a referee file through reviewrule on a mismatch they proposed

The recording is evidence of what the legacy system does, not of what is correct. A recorded answer that looks wrong is still the answer until the business owner rules on it.

How strong is the corpus?

Mutation testing answers that question directly. A tool such as Stryker, the reference choice for TypeScript, makes small changes to your code, one at a time (flip >= to >, replace a condition with false, swap + for -), and runs your tests against each changed copy, called a mutant. A mutant your tests fail on is killed. One they pass on survived: the tests could not tell the broken code from the real code.

Plan the change with your agent

Open your agent in the project root and switch to Plan mode (Shift+Tab, /plan, or the mode picker). Ask it to set up Stryker mutation testing against the intake rule files and the differential golden corpus:

We want to configure Stryker with the Vitest runner to mutate our intake rule files in src/features/intake/ against tests/differential and fail below a 100 mutation score. Ask me anything unclear before writing config.

What to check in the plan before you approve

  • stryker.config.json uses a glob (src/features/intake/**/*.ts) while excluding store.ts (the database adapter) and triage/, rather than hardcoding individual filenames.
  • The test runner runs only the golden corpus in tests/differential so the mutation score measures the human-approved corpus.
  • thresholds.break is set to 100, because in small rule files every surviving mutant is either a missing corpus case or an equivalent mutant requiring a formal ruling.
  • The initial run does not modify source code or add tests yet; it lists each surviving or uncovered mutant with its file, line, and mutation.

If the first plan lists only score.ts and submit.ts by name instead of a directory glob, correct it before approving:

Use a src/features/intake/**/*.ts glob with exclusions for store.ts and triage/ so a new rule file cannot escape mutation testing.

Here is the reference stryker.config.json from gov-dev-reference-app (compare with what your agent writes; you do not type it by hand):

{
"testRunner": "vitest",
"mutate": ["src/features/intake/**/*.ts", "!src/features/intake/store.ts", "!src/features/intake/triage/**"],
"vitest": { "dir": "tests/differential" },
"reporters": ["clear-text", "json"],
"coverageAnalysis": "perTest",
"thresholds": { "high": 100, "low": 90, "break": 100 }
}

The glob matters: listing two files by name would let anyone move a rule into a third file the tool never mutates. Stryker’s documentation is plain about break: “mutation score < break: Error! Stryker will exit with exit code 1, indicating a build failure” (Stryker configuration). Run against lesson 22’s ten cases, it reports:

[NoCoverage] ArithmeticOperator
src/features/intake/score.ts:6:26
- if (diff > 0.5) return floor + 1;
+ if (diff > 0.5) return floor - 1;
[Survived] ConditionalExpression
src/features/intake/score.ts:6:7
- if (diff > 0.5) return floor + 1;
+ if (false) return floor + 1;
File | total | covered | # killed | # timeout | # survived | # no cov |
score.ts | 93.33 | 96.55 | 28 | 0 | 1 | 1 |
submit.ts | 100.00 | 100.00 | 15 | 0 | 0 | 0 |
Final mutation score 95.56 under breaking threshold 100, setting exit code to 1 (failure).

The two findings are one gap. Every case in the corpus has a wait credit (daysWaiting / 20) of 0, 0.5, 1, 1.5, 2.5 or 10: whole numbers and exact halves. Lesson 22 chose the halves on purpose to settle how VBScript rounds a tie. But no case has a fraction above one half, so the line if (diff > 0.5) return floor + 1 never runs, and deleting it would pass all ten cases. That is the kind of gap a ten-case corpus always has, and it is invisible until something measures it.

Stryker measures the golden corpus only, in tests/differential/: submitIntake() against a recording store. The property replay below lives in tests/parity/ on purpose, so the mutation score keeps answering one question: how strong are the cases a person approved. It says nothing about the route handler, the legacy page or the database. Those are what the bench is for.

Agents propose inputs, legacy supplies answers

Still in Plan mode, ask the agent to read the mutation report alongside modernization/rules.md and propose candidate inputs without pre-filling the legacy answers:

Read modernization/rules.md, unknowns.md, and the Stryker report, and propose synthetic candidate inputs in modernization/candidates/ for each surviving mutant and unprobed rule boundary. Do not include expected outputs; our legacy recorder supplies those.

Check that each candidate file modernization/candidates/<id>.json records only id, synthetic input, exercises (R-ids), and targets (which mutant or boundary it probes, such as wait-credit fractions above and below 0.5 and householdSize: 4), with no expected output field.

Here are the three reference candidates in modernization/candidates/ from gov-dev-reference-app (compare with your agent’s proposed inputs; you do not type them by hand):

[
{ "id": "c11-wait-above-half", "input": { "caseType": "ROUTINE", "householdSize": 3, "daysWaiting": 11 }, "exercises": ["R-04"], "targets": "score.ts:6 ConditionalExpression, ArithmeticOperator (R-04: wait credit, rounded)" },
{ "id": "c12-wait-below-half", "input": { "caseType": "ROUTINE", "householdSize": 3, "daysWaiting": 5 }, "exercises": ["R-04"], "targets": "boundary: fraction 0.25" },
{ "id": "c13-household-four", "input": { "caseType": "ROUTINE", "householdSize": 4, "daysWaiting": 0 }, "exercises": ["R-03"], "targets": "boundary: one below the household-of-five point (R-03)" }
]

A pipeline job records the candidates against the legacy copy, under its own credentials. Lesson 22’s record-golden.ts reads modernization/golden/cases.json; give it an input-path argument (one line) so the job can point it at the candidates instead. A maintainer reads each recorded answer and moves the case into modernization/golden/ through a reviewed pull request. With c11 promoted, the same Stryker run reports 45 of 45 mutants killed, a score of 100, exit 0.

Generated inputs, recorded once

Hand-picked cases cover the boundaries someone thought of. Property-based testing covers the ones nobody did: you describe the input domain, the library generates inputs from it, and when one fails it shrinks it to the smallest input that still fails. fast-check is the reference choice for TypeScript (fast-check).

The domain matters as much as the property. The legacy page runs CInt() on two fields (submit.asp:6-7); lesson 22 recorded the runtime error on a bad value as U-05, and CInt only holds a 16-bit integer, so generated inputs stay inside what the old page accepts without an error. Otherwise the run reports U-05 a thousand times and parity never (reference implementation from gov-dev-reference-app for comparison with what your agent builds):

// tests/parity/arbitraries.ts — the input domain the legacy page accepts without a runtime error.
import fc from 'fast-check';
export const PROPERTY_SEED = 20261105;
export const PROPERTY_RUNS = 200;
export const intakeInput = fc.record({
caseType: fc.constantFrom('ROUTINE', 'EMERGENCY', 'emergency', ' EMERGENCY ', 'Routine'),
householdSize: fc.integer({ min: 1, max: 12 }), // CInt overflows past 32767 (U-05); stay well inside
daysWaiting: fc.integer({ min: 0, max: 400 }),
});
export const sampleInputs = () => fc.sample(intakeInput, { seed: PROPERTY_SEED, numRuns: PROPERTY_RUNS });

The legacy system is slow, it writes rows, and it lives inside the boundary, so the property run is split in two:

  • Recorded, in every build. The bench runner generates the 200 inputs for the fixed seed, posts each to the legacy copy once, and commits inputs and observations to modernization/golden/property-expected.json. The replay test below, in tests/parity/, compares the new code to that recording in process, in milliseconds, with no legacy server. It also regenerates the inputs and fails if they no longer match the recording, so a changed generator cannot quietly replay a stale file.
  • Live, nightly. A fresh seed, logged with the run, generates new inputs and posts them to both systems on the bench. A failure is reported with its seed and shrink path, which replay it exactly. The shrunk input becomes a candidate, and the loop above records it.

An approved change (see the rulings below) is the one difference allowed to stay, and only for the exact input, field and pair of values its ruling covers. Both the corpus test and the replay pass their mismatches through one filter in tests/differential/approved.ts, so every other comparison stays strict (reference implementation from gov-dev-reference-app):

// tests/differential/approved.ts — an approved change passes only for the exact input, field and pair of
// values its ruling covers; every other comparison stays strict.
import { readFileSync } from 'node:fs';
export type Approved = { ruling: string; input: unknown; path: string; legacy: unknown; next: unknown };
export const approved: Approved[] = JSON.parse(readFileSync('modernization/golden/approved-changes.json', 'utf8'));
export function withoutApproved<M extends { path: string; legacy: unknown; next: unknown }>(mismatches: M[], input: unknown): M[] {
const key = JSON.stringify(input);
return mismatches.filter((m) => !approved.some((a) =>
a.path === m.path && JSON.stringify(a.input) === key && Object.is(a.legacy, m.legacy) && Object.is(a.next, m.next)));
}

The list it reads, modernization/golden/approved-changes.json, is under a referee path, and the ruling checker below fails if an approved ruling has no entry in it.

Wire both folders into one script, "test:diff": "vitest run tests/differential tests/parity". One more expected red: upgrading fast-check can change what a seed generates, so the first replay test fails after a bump until the bench re-records the set. The replay calls runNew() with only an input and an actor, so lesson 23’s harness takes Pick<Case, 'input' | 'actor'>, the two fields it uses.

REFERENCE TEST tests/parity/property-replay.test.ts: in-process replay of recorded property inputs open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
tests/parity/property-replay.test.ts
import { readFileSync } from 'node:fs';
import { describe, expect, it } from 'vitest';
import { diffObservations, runNew } from '../differential/harness';
import { tolerances } from '../differential/tolerances';
import { withoutApproved } from '../differential/approved';
import { sampleInputs, PROPERTY_SEED, PROPERTY_RUNS } from './arbitraries';
const rec = JSON.parse(readFileSync('modernization/golden/property-expected.json', 'utf8'));
describe('recorded property set: legacy observations vs new, in process', () => {
it('was recorded from the generator as it stands', () => {
expect({ seed: rec.seed, numRuns: rec.numRuns }).toEqual({ seed: PROPERTY_SEED, numRuns: PROPERTY_RUNS });
expect(rec.cases.map((c: any) => c.input)).toEqual(sampleInputs());
});
it('every recorded input matches', async () => {
const bad: string[] = [];
for (const c of rec.cases) {
const mm = withoutApproved(diffObservations(c.legacy, await runNew({ input: c.input, actor: 'qa.caseworker' }), tolerances), c.input);
if (mm.length) bad.push(`${JSON.stringify(c.input)}: ${mm.map((m) => `${m.path} legacy=${m.legacy} new=${m.next}`).join('; ')}`);
}
expect(bad, bad.slice(0, 3).join('\n')).toEqual([]);
});
});

On the bench, a few rules keep a live run honest. Run inputs one at a time, because both systems are writing real rows. Read each side’s rows by the CaseId its response returned, every row that carries it, so an extra or missing audit row is a difference rather than noise. Assert that every audit row carries the response’s CaseId, the relationship lesson 23’s in-memory harness had no way to check. Turn off everything that leaves the building on both restores before the run: the export job, mail and any queue. And bound the run by time, not by a number someone liked. When both restores come from the same backup and replay in the same order, the two systems usually assign the same CaseId; a rollback, or the identity cache jumping after an unclean restart, burns values on one side and they drift apart, so the bench compares IDs under a tolerance whose reason says exactly that.

Rows accumulate in the restores across a night’s run. The intake rules are stateless, so that does not change any answer. A rule that reads earlier rows, such as duplicate detection, would need a fresh restore per trial, and the nightly refresh is the reset.

The referee is out of the agent’s reach

The paths that decide pass or fail are listed in one file:

REFERENCE CONFIG modernization/referee-paths.txt: protected paths that decide pass or fail open reference hide reference Reference file from gov-dev-reference-app; open to inspect or compare with what your agent builds.
modernization/referee-paths.txt
# Files that decide pass or fail. The parity agent may read them and never change them.
tests/differential/
tests/parity/
modernization/rules.md
modernization/unknowns.md
modernization/golden/
modernization/scripts/
modernization/parity/rulings/
modernization/parity/equivalent-mutants.txt
modernization/referee-paths.txt
modernization/owner-teams.txt
modernization/variant.json
modernization/supported-variants.json
scripts/capstone/
scripts/parity/
scripts/verify-cold.sh
stryker.config.json
vitest.config.ts
evidence/parity-report.json
evidence/evidence-manifest.json
.claude/
package.json
# Your host's owners file and pipeline definition (lesson 37), one pair:
.github/CODEOWNERS
.github/workflows/
# GitLab: CODEOWNERS and .gitlab-ci.yml · Bitbucket: .bitbucket/CODEOWNERS and bitbucket-pipelines.yml · Azure DevOps: azure-pipelines.yml

The control that holds is review, not the agent’s settings. On GitHub, a CODEOWNERS file assigns owners to paths, and with “Require review from Code Owners” turned on in branch protection, “any pull request that affects code with a code owner must be approved by that code owner before the pull request can be merged” (About code owners). Azure DevOps has no CODEOWNERS file; its equivalent is the “Automatically included reviewers” branch policy with a path filter, set to Required (branch policies). The list is longer than people expect, because anything that changes the verdict is the referee: the tests that compare as well as the harness, the scripts in package.json that decide which tests run, the report and the evidence manifest, the protection list itself, CODEOWNERS and the pipeline definitions. Leave out one and the protection is one edit away from gone. Identify the agent by the account it commits and opens pull requests as, never by a branch name, which anyone can choose. That needs the agent to run under its own bot or service account; an agent working in a developer’s CLI commits as that developer, and every account-based check fails open.

A check that the ownership is declared runs in every build:

REFERENCE CODE scripts/parity/check-referee.mjs: verify code-owner protection across referee paths open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// scripts/parity/check-referee.mjs — every referee path, and every file under it, is owned only by
// teams, never by the agent, so no change to it merges without a code owner's review.
// It reads files, so it proves ownership is DECLARED. Whether the host enforces it is a setting
// checked by a person (lesson 38's checklist). One adapter per code host (REPO_HOST):
// github (default) and bitbucket: CODEOWNERS, last matching line wins;
// gitlab: CODEOWNERS with [Sections]; ^[optional] sections require no approval;
// azure-devops: no owners file; the exported required-reviewer policies are checked instead.
// Usage: REPO_HOST=<host> node scripts/parity/check-referee.mjs [repo-root]
import { existsSync, readdirSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
const root = process.argv[2] ?? '.';
const at = (p) => join(root, p);
const HOST = process.env.REPO_HOST ?? 'github';
const AGENT_ACCOUNTS = ['@intake-parity-agent']; // the accounts the agent commits and opens PRs as
const HOSTS = {
github: { owners: ['.github/CODEOWNERS', 'CODEOWNERS', 'docs/CODEOWNERS'], pipeline: ['.github/workflows/'] },
gitlab: { owners: ['CODEOWNERS', 'docs/CODEOWNERS', '.gitlab/CODEOWNERS'], pipeline: ['.gitlab-ci.yml'] },
bitbucket: { owners: ['.bitbucket/CODEOWNERS'], pipeline: ['bitbucket-pipelines.yml'] },
'azure-devops': { owners: [], pipeline: ['azure-pipelines.yml'] },
};
const host = HOSTS[HOST];
if (!host) { console.error(`REPO_HOST must be one of: ${Object.keys(HOSTS).join(', ')}`); process.exit(1); }
const MUST_PROTECT = ['tests/differential/', 'tests/parity/', 'modernization/rules.md', 'modernization/unknowns.md',
'modernization/golden/', 'modernization/scripts/', 'modernization/parity/rulings/',
'modernization/parity/equivalent-mutants.txt', 'modernization/referee-paths.txt', 'modernization/owner-teams.txt',
'modernization/variant.json', 'modernization/supported-variants.json', 'scripts/capstone/',
'scripts/parity/', 'scripts/verify-cold.sh', 'stryker.config.json', 'vitest.config.ts', 'package.json',
'evidence/parity-report.json', 'evidence/evidence-manifest.json', '.claude/', ...host.owners.slice(0, 1), ...host.pipeline];
// a shrunken referee-paths.txt fails here
const lines = (f) => readFileSync(at(f), 'utf8').split('\n').map((s) => s.split('#')[0].trim()).filter(Boolean);
const problems = [];
const referee = lines('modernization/referee-paths.txt');
for (const p of MUST_PROTECT) if (!referee.includes(p)) problems.push(`referee-paths.txt is missing ${p}`);
// Which owners count as a team. GitHub writes teams as @org/team; the other hosts cannot be told
// apart from people by syntax, so they name their teams (or, on Azure DevOps, group ids) in a file.
const teams = existsSync(at('modernization/owner-teams.txt')) ? new Set(lines('modernization/owner-teams.txt')) : new Set();
const isTeam = (o) => (HOST === 'github' ? /^@[\w.-]+\/[\w.-]+$/.test(o) : teams.has(o));
const badOwners = (owners) => owners.length === 0 || owners.some((o) => !isTeam(o) || AGENT_ACCOUNTS.includes(o));
// `*` (the usual default line) covers everything; `/dir` and `/dir/` both cover the directory and its contents.
const covers = (pattern, path) => {
if (pattern === '*') return true;
const p = pattern.replace(/\/$/, ''), f = `/${path}`.replace(/\/$/, '');
return f === p || f.startsWith(`${p}/`);
};
const inside = (pattern, dir) => dir.endsWith('/') && pattern !== '*' && pattern.startsWith(`/${dir}`) && pattern.length > dir.length + 1;
if (HOST === 'azure-devops') {
// Export first, for the protected branch: az repos policy list --branch "$REFEREE_BRANCH" --output json > ci-out/branch-policies.json
const REQUIRED_REVIEWERS = 'fd2167ab-b0be-447a-8ec8-39368250530e'; // the policy type id
const f = 'ci-out/branch-policies.json';
if (!existsSync(at(f))) { console.error(`no ${f}: export the branch policies first`); process.exit(1); }
// A policy counts only if it applies to the protected branch (and, if REPO_ID is set, this repository).
const branch = `refs/heads/${process.env.REFEREE_BRANCH ?? 'main'}`.toLowerCase();
const inScope = (p) => (p.settings?.scope ?? []).some((s) =>
(s.repositoryId == null || !process.env.REPO_ID || s.repositoryId === process.env.REPO_ID) &&
(String(s.matchKind).toLowerCase() === 'prefix' ? branch.startsWith(String(s.refName).toLowerCase()) : String(s.refName).toLowerCase() === branch));
const policies = JSON.parse(readFileSync(at(f), 'utf8'))
.filter((p) => p.type?.id === REQUIRED_REVIEWERS && p.isEnabled && p.isBlocking && !p.isDeleted && inScope(p));
// Only exact paths and "dir/*" filters are read here; anything else needs a person.
const filterCovers = (pat, path) => {
const p = pat.startsWith('/') ? pat : `/${pat}`;
return p === `/${path}` || (p.endsWith('/*') && `/${path}`.startsWith(p.slice(0, -1)));
};
for (const p of policies) for (const pat of p.settings?.filenamePatterns ?? []) {
if (!/^\/?[\w./-]+(\/\*)?$/.test(pat)) problems.push(`policy ${p.id}: filter ${pat} is not an exact path or dir/*; check it by hand`);
}
for (const path of referee) {
const hits = policies.filter((p) => (p.settings?.filenamePatterns ?? []).some((pat) => filterCovers(pat, path)));
if (!hits.length) problems.push(`${path}: no enabled, blocking required-reviewer policy covers it`);
for (const p of hits) if (badOwners(p.settings?.requiredReviewerIds ?? [])) problems.push(`${path}: policy ${p.id} names reviewers that are not listed teams`);
}
} else {
const ownersFile = host.owners.find((f) => existsSync(at(f))); // the host reads the first one it finds
if (!ownersFile) { console.error(`no CODEOWNERS in ${host.owners.join(', ')}`); process.exit(1); }
if (ownersFile !== host.owners[0]) problems.push(`CODEOWNERS is at ${ownersFile}; keep it at ${host.owners[0]}, the path the referee list protects`);
// Rules grouped by section. GitHub and Bitbucket have one implicit, required section.
const rules = [];
let section = { name: '(none)', optional: false, defaults: [] };
for (const l of lines(ownersFile)) {
const head = HOST === 'gitlab' && l.match(/^(\^)?\[([^\]]+)\](?:\[\d+\])?\s*(.*)$/);
if (head) { section = { name: head[2], optional: !!head[1], defaults: head[3].split(/\s+/).filter(Boolean) }; continue; }
const [pattern, ...owners] = l.split(/\s+/);
rules.push({ pattern, owners: owners.length ? owners : section.defaults, section });
}
for (const r of rules) if (r.pattern !== '*' && (/[*?[\]!]/.test(r.pattern) || !r.pattern.startsWith('/'))) problems.push(`CODEOWNERS ${r.pattern}: only "*" and root-anchored paths are supported here`);
for (const path of referee) {
// Last matching line wins, within each section (GitLab evaluates every section).
// Sections with the same name are one section (GitLab merges them).
const effective = [...new Set(rules.map((r) => r.section.name))]
.map((n) => rules.filter((r) => r.section.name === n && covers(r.pattern, path)).at(-1)).filter(Boolean);
if (!effective.some((r) => !r.section.optional)) problems.push(`${path}: no required CODEOWNERS section covers it`);
for (const r of effective) if (badOwners(r.owners)) problems.push(`${path}: effective owners (${r.pattern}${r.section.name === '(none)' ? '' : ` in [${r.section.name}]`}) are not all teams, or include the agent`);
for (const r of rules.filter((x) => inside(x.pattern, path))) { // a narrower line can hand one file away
if (badOwners(r.owners)) problems.push(`${r.pattern}: inside referee path ${path}, owners are not all teams, or include the agent`);
}
}
if (HOST === 'bitbucket') console.log('NOTE Bitbucket has no required code-owner approval: the compensating check in lesson 37 enforces it');
}
// A "Stryker disable" comment removes a mutant from the score, the cheapest way to turn the mutation step
// green. Each one must be listed as "file:line M-nnnn", and that ruling must exist.
const allowed = new Set();
for (const entry of lines('modernization/parity/equivalent-mutants.txt')) {
const [where, id] = entry.split(/\s+/);
if (!/^M-\d{4}$/.test(id ?? '') || !existsSync(at(`modernization/parity/rulings/${id}.json`))) problems.push(`equivalent-mutants.txt ${where}: no ruling ${id ?? '(none given)'}`);
else allowed.add(where);
}
for (const f of readdirSync(at('src'), { recursive: true }).map(String).filter((f) => /\.(c|m)?(t|j)sx?$/.test(f))) {
readFileSync(at(`src/${f}`), 'utf8').split('\n').forEach((line, i) => {
if (/stryker\s+disable/i.test(line) && !allowed.has(`src/${f}:${i + 1}`)) problems.push(`src/${f}:${i + 1}: Stryker disable with no ruling in equivalent-mutants.txt`);
});
}
if (problems.length) { console.error(problems.join('\n')); process.exit(1); }
console.log(`referee (${HOST}): ${referee.length} paths, each owned only by teams`);

It reads files, so it runs on a cold clone. It cannot see whether branch protection is switched on in the hosting service; that is one line in the reviewer’s checklist in lesson 38. REPO_HOST picks the adapter, and the hosts differ in ways that matter here:

  • GitHub and Bitbucket read one owners file, and the last matching line wins. GitHub writes teams as @org/team; Bitbucket cannot tell a team from a person by syntax, so its teams are named in modernization/owner-teams.txt, itself a referee file. Bitbucket also has no setting that requires a code owner’s approval: the script says so, and lesson 37 adds the compensating check.
  • GitLab owners files have sections. Each section is evaluated on its own, and a section written ^[Name] is optional: it requires no approval. So a referee path covered only by an optional section fails, and an owner who is not a listed team fails in any section that covers the path.
  • Azure DevOps has no owners file. A pipeline step exports the branch policies for each protected branch (az repos policy list --branch "$REFEREE_BRANCH" --output json > ci-out/branch-policies.json, run once per protected branch with REFEREE_BRANCH set), and the script requires an enabled, blocking required-reviewer policy that applies to that branch (its scope) (type fd2167ab-b0be-447a-8ec8-39368250530e in the policy configurations API) whose path filter covers each referee path and whose reviewers are listed groups. It reads only exact paths and dir/* filters; anything fancier is reported for a person to check.

The adapters have tests of their own in the runnable reference (tests/unit/check-referee.test.ts, 16 passing), including the bypass each host allows: a narrower line that hands one file to the agent, a * default owned by a person, an optional GitLab section, a non-blocking Azure DevOps policy, a policy scoped to another branch, and a reviewer who is not a listed group. The minimum list is written into the script, which is itself a referee file, so shortening referee-paths.txt fails the build. The pull-request pipeline adds one check that needs a diff: a pull request opened by the agent’s account fails if it touches any referee path at all.

The agent’s own settings are a convenience on top, so it stops before trying instead of being refused at review. In Claude Code that is a deny rule in the committed .claude/settings.json; deny rules are evaluated before allow rules, and “an allow rule can’t carve an exception out of a deny rule” (Claude Code permissions):

REFERENCE CONFIG .claude/settings.json: editor deny list for referee paths open reference hide reference Reference file pattern for your workspace agent; open to inspect or compare. You do not type or paste this by hand.
{
"permissions": {
"deny": [
"Edit(/tests/differential/**)",
"Edit(/tests/parity/**)",
"Edit(/modernization/rules.md)",
"Edit(/modernization/unknowns.md)",
"Edit(/modernization/scripts/**)",
"Edit(/modernization/referee-paths.txt)",
"Edit(/scripts/parity/**)",
"Edit(/stryker.config.json)",
"Edit(/vitest.config.ts)",
"Edit(/scripts/verify-cold.sh)",
"Edit(/azure-pipelines.yml)",
"Edit(/package.json)",
"Edit(/evidence/**)",
"Edit(/.claude/**)",
"Edit(/.github/**)",
"Edit(/modernization/golden/**)",
"Edit(/modernization/parity/rulings/**)",
"Edit(/modernization/parity/equivalent-mutants.txt)"
]
}
}

Keep the list in step with referee-paths.txt. Edit(...) rules cover the editor tools. They do not stop a shell command such as sed -i or a redirect from writing the same file, which is why the code-owner review, and not this file, is the control. Other agent CLIs have their own equivalents; the same limit applies to each.

Many agents, one bench: worktrees, a headless job and review gates

The bench so far has one agent in one checkout. A rebuild has many candidate batches and many modules, and three working rules keep more agents from weakening the referee.

Parallel work goes in worktrees. A git worktree is a second checkout of the same repository on its own branch, so two agents cannot overwrite each other’s files. Give one worktree to each candidate batch or each module (lesson 22’s module list is the split), and let each agent propose and fix inside its own tree. In Claude Code, claude --worktree <name> (or -w) creates .claude/worktrees/<name>/ on branch worktree-<name>; in Codex CLI, codex --worktree starts a new interactive session the same way. The pattern itself is tool-agnostic and Module 19 lesson 5 (parallel sessions in git worktrees, and when one session or a subagent is enough) covers the choice. A worktree separates files, not authority: each tree still reads the same referee-paths.txt, still runs as the agent’s own account, and every pull request from every tree meets the same code owners.

Candidate generation can run headless in CI. The prompt from “Agents propose inputs” does not need a person at a terminal. A pipeline job runs it read-only, with no cloud credentials and no database access, and asks for the candidates in a shape a schema can check:

Terminal window
claude -p "$(cat modernization/prompts/propose-candidates.md)" \
--bare --output-format json \
--json-schema "$(cat modernization/candidates.schema.json)" \
--permission-mode dontAsk --allowedTools "Read Grep Glob" \
> ci-out/candidates.envelope.json

--bare is the recommended mode for scripted calls: it skips hooks, skills, plugins and CLAUDE.md, so the prompt file has to carry every rule the agent needs, and it needs an API key from the job’s secret store, not a login. The job has no other credentials. The recorder job from the section above, which does hold the legacy copy’s credentials, runs separately and reads only the validated file. Other tools take the same shape with different spellings: codex exec --json --output-schema <file> -o <file> for Codex CLI and agy -p --output-format json --json-schema ... for Antigravity; lesson 19 gives all four.

The referee checks the artifact, never the exit status. agy -p soft-denies an action it was not pre-approved to take and still exits 0, and --bare drops the project memory a rule may live in, so a run that read nothing can look identical to a clean run. After the model call, a deterministic script validates candidates.json against the schema, rejects any expected or output field (the agent never writes an answer), and fails when the file is empty or names a rule id that is not in rules.md. That is the check-review pattern from lesson 19, with the candidate file in place of the findings file.

Review gates sit on the two moves that change the verdict. Promoting a recorded candidate into the golden corpus and changing a tolerance both go through a pull request whose owner is a named person, never the account that proposed it. A tolerance change adds a second step: a council review (Module 19 lesson 8, several models reviewing the same diff before a risky change ships) reads the diff and the mismatch evidence first, and its findings go on the pull request for the named person. The council advises; the named person rules, as in the next section.

Mismatch rulings

Every mismatch the bench finds gets an id, and its evidence is a hash of the input and both observations, so a ruling binds to exactly what was seen; if the next run shows different values, the old ruling no longer counts. The agent writes a proposal in modernization/parity/proposals/. A named person, not the proposer, writes the ruling in modernization/parity/rulings/, which is a referee path: if both lived in one file, the agent could write the ruling too. Here are the reference proposal and ruling files from gov-dev-reference-app (for comparison; you do not type them by hand):

// modernization/parity/proposals/M-0001.json (the agent's)
{
"evidence": "<sha256 of [input, legacy, new]>",
"by": "intake-parity-agent",
"classification": "regression",
"why": "score.ts:6 no longer rounds fractions above one half up; legacy Round(0.55) = 1, new = 0. Rule R-04."
}
// modernization/parity/rulings/M-0001.json (a person's, merged with the code owner's review)
{ "evidence": "<the same sha256>", "decision": "regression: fix the new code", "by": "j.rivera", "on": "2026-11-05" }
DecisionWhat happens nextMay it still be a mismatch at cutover?
regression: fix the new codethe agent may propose the fix; the case joins the corpus once recordedno: the next report must not contain it
legacy defect: preservethe new code changes to match the legacy behaviorno
legitimate difference: tolerancea tolerance with an exact path and a reason, approved by the rule’s ownerno: the tolerance absorbs it
approved changethe business owner changes the rule in rules.md; the recorded legacy answer stays as recorded, because it is what the old system did; the ruling names the rule and, in approves, the exact input, field and values, and the same entry goes in approved-changes.jsonyes, and only this one
unknown: blocks cutoveran entry in unknowns.md with an ownerno: the route does not flip

A ruling is a disposition, not a fix. The checker below treats it that way: a regression that is ruled but still in the report still blocks.

The checker fails the build on any mismatch without a ruling, on a ruling made against different evidence or by the proposer, on any ruled mismatch other than an approved change that is still in the report, on a report whose compared count does not match the corpus and the recorded property set (fresh nightly inputs are counted separately, in fresh), and on any entry in approved-changes.json that is not exactly what an approved ruling approves, whether or not that mismatch is in today’s report:

REFERENCE CODE scripts/parity/check-rulings.mjs: verify every mismatch has a human ruling bound to its evidence hash open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// scripts/parity/check-rulings.mjs — usage: node check-rulings.mjs evidence/parity-report.json
// The report is written by the bench runner and is a referee path. It covers the live nightly run; the
// corpus and the recorded property set are re-checked in every build by test:diff, so an edited report
// cannot hide a difference there. Every mismatch in it needs a ruling
// by a named person, bound to that mismatch's exact evidence, and only an approved change may still
// be a mismatch at cutover: everything else has to disappear from the next report.
import { createHash } from 'node:crypto';
import { existsSync, readFileSync } from 'node:fs';
const STAYS = 'approved change'; // rules.md changed on the owner's decision
const GOES = ['regression: fix the new code', 'legacy defect: preserve', 'legitimate difference: tolerance'];
const BLOCKS = 'unknown: blocks cutover';
const report = JSON.parse(readFileSync(process.argv[2], 'utf8'));
const read = (f) => (existsSync(f) ? JSON.parse(readFileSync(f, 'utf8')) : null);
const evidenceHash = (m) => createHash('sha256').update(JSON.stringify([m.input, m.legacy, m.next])).digest('hex');
const rules = readFileSync('modernization/rules.md', 'utf8');
const approved = read('modernization/golden/approved-changes.json') ?? [];
const problems = [];
// An empty mismatch list proves nothing unless the run compared what it should have.
const corpus = read('modernization/golden/cases.json')?.length ?? 0;
const property = read('modernization/golden/property-expected.json')?.cases?.length ?? 0;
if (report.compared !== corpus + property) problems.push(`report compared ${report.compared}, expected ${corpus} corpus + ${property} property inputs`);
// Fresh nightly inputs are counted separately, in report.fresh; their mismatches are in the same list.
for (const m of report.mismatches) {
const proposal = read(`modernization/parity/proposals/${m.id}.json`);
const ruling = read(`modernization/parity/rulings/${m.id}.json`);
if (!proposal) problems.push(`${m.id}: no proposal on file; every mismatch has one, so proposer and ruler can be told apart`);
if (!ruling) { problems.push(`${m.id}: no ruling${proposal ? ` (proposal on file: ${proposal.classification})` : ''}`); continue; }
if (ruling.evidence !== evidenceHash(m)) problems.push(`${m.id}: ruling was made on different evidence; rule again`);
if (!/^[a-z]+\.[a-z]+$/.test(ruling.by ?? '')) problems.push(`${m.id}: "by" is not first.last (a format check; review is the real control)`);
if (ruling.by === proposal?.by) problems.push(`${m.id}: the proposer cannot rule`);
if (!/^\d{4}-\d{2}-\d{2}$/.test(ruling.on ?? '')) problems.push(`${m.id}: ruling date missing`);
if (ruling.decision === STAYS) {
if (!/^R-\d+$/.test(ruling.rule ?? '') || !rules.includes(`| ${ruling.rule} |`)) problems.push(`${m.id}: an approved change must name a rule that is in rules.md`);
if (!approved.some((a) => a.ruling === m.id)) problems.push(`${m.id}: approved change missing from modernization/golden/approved-changes.json`);
}
else if (GOES.includes(ruling.decision)) problems.push(`${m.id}: ruled "${ruling.decision}" but still in the report; fix it and re-run`);
else if (ruling.decision === BLOCKS) problems.push(`${m.id}: open unknown blocks cutover`);
else problems.push(`${m.id}: unknown decision "${ruling.decision}"`);
}
// Every approved-change entry, present in this report or not, must be exactly what an approved ruling approves.
for (const a of approved) {
const ruling = read(`modernization/parity/rulings/${a.ruling}.json`);
const { ruling: _id, ...entry } = a;
if (!ruling || ruling.decision !== STAYS) problems.push(`approved-changes.json ${a.ruling}: no approved-change ruling with that id`);
else if (JSON.stringify(ruling.approves) !== JSON.stringify(entry)) problems.push(`approved-changes.json ${a.ruling}: entry differs from what the ruling approves`);
}
if (problems.length) { console.error(problems.join('\n')); process.exit(1); }
console.log(`rulings: ${report.compared} inputs compared, ${report.mismatches.length} approved change(s), nothing unruled`);

The decision behind all of this is a rule, so record it in rules.md, with its decision file modernization/decisions/D-02-parity-rulings.md holding the decision on line 4:

IDRuleCitationSide effects
R-11a parity mismatch blocks cutover until a named person other than its proposer rules on its exact evidencedecisions/D-02-parity-rulings.md:4none
Its test of record is the checker’s own test in tests/parity/check-rulings.test.ts (reference excerpt from gov-dev-reference-app), which runs the script in a temporary directory against planted files:
// tests/parity/check-rulings.test.ts (excerpt): the test of record for R-11
it('a proposal alone is not a ruling', () => {
const r = run(base); // report with one mismatch, the agent's proposal, no ruling
expect(r.status).toBe(1);
expect(r.stderr).toContain('M-0001: no ruling');
});
it('a ruled regression still in the report blocks', () => { /* ... */ });
it('an approved change on its own evidence passes', () => { /* ... */ });
it('an emptied report fails on the compared count', () => { /* ... */ });
it('a ruling with no proposal on file, or naming a rule not in rules.md, fails', () => { /* ... */ });
it('an approved-change entry with no matching ruling fails, even when the report is clean', () => { /* ... */ });

Add it to lesson 24’s tested-elsewhere.json as "R-11": { "file": "tests/parity/check-rulings.test.ts", "name": "a proposal alone is not a ruling" }. The by field is a format check; that the named person really ruled comes from the code owner’s review of the ruling file, which lesson 38’s checklist asks the reviewer to confirm.

The nightly run ties it together. It refreshes both restores from the latest backup, turns off their outbound jobs, replays the corpus and the recorded property set, runs a bounded live property run with a fresh logged seed, and writes evidence/parity-report.json with the number of inputs compared and the mismatches (a deterministic summary; seeds, timings and the snapshot id go in the run log beside it). The report changes only through a reviewed pull request from the bench runner’s account. It covers the live run; an edited report cannot hide a difference in the corpus or the recorded property set, because test:diff re-checks both in every build. The agent then proposes classifications for anything new, in its own step, with no database access. Lesson 19’s scheduled pipeline carries the job; lesson 38 checks the report.

Watch it work

SESSION A planted fault the corpus misses, found, ruled and blocked
  1. Typed in the terminal: claude
  2. Mode:Plan mode
  3. You type in the session: Someone simplified bankersRound and deleted the 'diff > 0.5' line. Plan how we check whether the golden corpus or the recorded property replay catches it before changing any code.
  4. The tool replies: I will run tests/differential/intake.diff.test.ts first to see if the 10-case golden corpus catches the missing branch, and then run tests/parity/property-replay.test.ts against the 200 recorded property inputs. Should I proceed with those two test runs?
  5. You type in the session: Yes. Run the golden corpus, then the recorded property replay, and report.
  6. The tool replies: npx vitest run tests/differential/intake.diff.test.ts Tests 10 passed (10) npx vitest run tests/parity/property-replay.test.ts FAIL every recorded input matches {"caseType":"emergency","householdSize":12,"daysWaiting":17}: response.score legacy=8 new=7; intake.score legacy=8 new=7 {"caseType":"EMERGENCY","householdSize":10,"daysWaiting":15}: response.score legacy=8 new=7; intake.score legacy=8 new=7 {"caseType":"ROUTINE","householdSize":1,"daysWaiting":53}: response.score legacy=5 new=4; intake.score legacy=5 new=4 13 of 200 recorded inputs differ. The ten golden cases cannot see this change: none has a wait-credit fraction above one half. Every failing input does (0.85, 0.75, 0.65 ...).
  7. You type in the session: Replay the failure with the property run's seed, 20261105, and give me the shrunk counterexample.
  8. The tool replies: Property failed after 4 tests { seed: 20261105, path: "3:0:0:2:0", endOnFailure: true } Counterexample: [{"caseType":"ROUTINE","householdSize":1,"daysWaiting":11}] Shrunk 4 time(s) Wrote modernization/parity/proposals/M-0001.json: classification regression, rule R-04. Wrote candidate c11-wait-above-half (daysWaiting 11). No expected output written; the recorder fills it.
  9. You type in the session: Quickest way to green: add a tolerance for response.score and intake.score.
  10. The tool replies: I can't edit tests/differential/tolerances.ts: project settings deny it, and it is a referee path owned by @agency/intake-business-owners. Widening a tolerance on a business field would also fail tolerances.test.ts from lesson 23. The fix is in the code: restore the 'diff > 0.5' branch in score.ts. Shall I?
  11. You type in the session: Yes. Then run the replay, and the ruling check against the last nightly report.
  12. The tool replies: Restored score.ts:6. property-replay: 2 passed node scripts/parity/check-rulings.mjs evidence/parity-report.json M-0001: no ruling (proposal on file: regression) exit 1 That failure is correct: I proposed, so I cannot rule. M-0001 needs a ruling from a named person, and the next nightly run, with the fix in, must no longer report it.

Illustrative session; exact output varies by tool and agency configuration.

The Stryker, Vitest and fast-check output in this lesson comes from running this code, type-checked, with the pinned majors from lesson 02; a stand-in for the ASP page supplied the legacy observations, since a classic ASP host was not part of that run. To reproduce the replay steps you need one recording from your legacy copy first (the bench runner’s job); without one, the corpus and mutation steps still run.

Prove it

Run these in order; the order matters, because promoting c11 changes what the corpus can see.

1. The corpus is measured. On lesson 22’s ten cases, npx stryker run exits 1 with a mutation score of 95.56 and lists the two score.ts:6 mutants.

2. The plant the corpus misses. Still on the ten cases, delete if (diff > 0.5) return floor + 1; from score.ts. Expected: intake.diff.test.ts passes, 10 of 10, and tests/parity/property-replay.test.ts fails, listing inputs whose wait credit has a fraction above one half (13 of the 200 recorded for seed 20261105). Restore the line.

3. The gap closes. Record and promote c11-wait-above-half. Stryker reports 45 of 45 killed, exit 0, and the same plant now fails the corpus too.

4. The ruling check. Run npx vitest run tests/parity/check-rulings.test.ts: six tests pass, covering a proposal with no ruling, a ruled regression still in the report, an approved change on its own evidence, a report whose compared count was zeroed, a ruling with no proposal that cites a rule not in rules.md, and an approved-change entry with no matching ruling (or one that differs from what its ruling approves).

5. The referee. node scripts/parity/check-referee.mjs passes on a correct CODEOWNERS. Then plant each of these, one at a time, and watch it fail with the path named:

PlantMessage
append /modernization/golden/property-expected.json @intake-parity-agentinside referee path modernization/golden/, owners are not all teams, or include the agent
append /tests/parity/ @agency/intake-maintainers @j-randomeffective owners (/tests/parity/) are not all teams
delete package.json from referee-paths.txtreferee-paths.txt is missing package.json
add a file under src/features/intake/ containing // stryker disable allStryker disable with no ruling in equivalent-mutants.txt
list src/features/intake/score.ts:6 in equivalent-mutants.txt with no ruling idno ruling (none given)

The last one matters most: that comment would take the mutation score to 100 by hiding mutants, not by killing them.

KNOWLEDGE CHECK

The agent proposes 40 new cases and, to save a step, also writes the expected outputs it computed from rules.md. All 40 pass against the new code. What do they prove?

KNOWLEDGE CHECK

The mutation step is red with one survivor. A teammate suggests lowering thresholds.break from 100 to 95. What is the right response?

Practice status — agents widening parity evidence without touching the referee

PracticeStatusAlso called
mutation testing to measure the corpus, with a break thresholdstrong optionalmutation analysis; test-suite adequacy
property-based inputs with a logged seed and shrinkingstrong optionalgenerative testing; fuzzing (for valid inputs)
recorded property set replayed in every build, live run nightlyemergingrecord and replay; golden master at scale
agent proposes inputs, legacy supplies answersemergingoracle separation
referee files owned by code owners with required reviewcommon baselineCODEOWNERS; required reviewers; protected paths
parallel agents in separate git worktrees, one per candidate batch or modulecommon baselineparallel sessions; branch-per-agent
headless candidate job: read-only, no cloud credentials, schema-checked artifactemergingscripted agent run; CI agent step
council review before a tolerance change, ruling by a named personemergingmulti-model review gate; four-eyes change control
every mismatch ruled by a named person on its exact evidence, and only approved changes left at cutoveremergingdefect disposition; discrepancy report

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Key takeaway

The ten golden cases prove ten things. Mutation testing shows which rule branches they never reach, an agent proposes the inputs that reach them, and only the legacy system says what the answer is. Generated inputs are recorded once and replayed in every build, with a live run against both systems at night. The agent never edits the referee: code owners hold those files, and a check proves the ownership is declared. Every mismatch waits for a named person’s ruling on its own evidence. Lesson 26 adds the first behavior with no legacy answer at all, a model call, and the bench keeps guarding every rule around it.

LEADERSHIP DECISION
no route flips while any mismatch is unruled; referee files change only with their code owner's review, and the agent's account owns none of them
PRACTITIONER ACTION
add the mutation step and the recorded property replay to CI, run the live bench nightly, and route every new mismatch to a named person for a ruling
SUCCESS MEASURE
contract or revenue risk avoided: no surviving mutant in the rule files, and no mismatch at cutover except an approved change with its rule named

Legend: LEADERSHIP DECISION = what a leader decides · PRACTITIONER ACTION = what the engineer does next · SUCCESS MEASURE = how you know, in calendar time, assessor-hours or cost, contract or revenue risk, or audit finding avoided

Search lessons