Applied AI-Accelerated Government Development

Capstone: Modernize an Intake Module, Evidence Attached

Updated

Advanced

What you'll learn

~110 min
  • Assemble the rules table, differential tests, the database contract, the parity bench, a model-backed triage, an SBOM (software bill of materials: the list of components in your build), an evidence manifest, a control statement and operator notes into one pull request
  • Write one documented command that a cold runner (someone who did not build it) runs to reproduce every non-live check
  • Keep the live endpoint test, the database tests and the live parity bench as separately named environment tests, never silently mocked
  • Declare your variant (cloud, boundary profile, code host, legacy stack, app type) in a file a referee checks, so extensions add obligations and never remove one
  • Report two outcomes that are never combined: Local exercise and Deployment evidence
  • Make every shipped behavior traceable to a rule, a test, an evidence item and a named human owner, and check that mechanically
ℹLeadership brief

What it is: one pull request that carries the modernized intake route together with the evidence that it behaves like the old one, and a single command that lets someone outside the team re-run every check that does not need a live service.

What it buys: a review that reads proof instead of promises, and a handoff that does not depend on the people who wrote it being in the room.

What to fund: reviewer time for the cold run, and a named owner and calendar slot for the environment tests no script can replace: the live model endpoint, the database checks and the nightly parity bench.

Before the detail — Artifact: one pull request with ten parts (below), a declared variant and a script named verify:cold. Acceptance test: a cold runner follows the README, runs that one command, and reads two separate outcomes, “Local exercise: COMPLETE” with no question to you, and a “Deployment evidence” line that stays INCOMPLETE until named people attach their runs. Status of what follows: reusable guidance. The cold-runner device is the one Module 11’s capstone uses on a small build (Prove It: a stranger runs your checks, and the failures become the gap log); here it carries a government-grade change.

You built the pieces across lessons 20 to 27, and the variant you declare below adds the lessons that match your situation: the platform (lessons 34 to 36: Azure Government, AWS GovCloud, Google Assured Workloads), the code host (lesson 37), the legacy stack (the Lane 5 field guides, lesson 28 and lesson 29, and lesson 22) and the app type (lessons 30 to 33). The baseline is the intake route on a classic ASP and T-SQL stack running locally, and it always runs in full. This lesson assembles them so a reviewer who was not in any of those lessons can approve or reject the change on evidence.

ℹNames assumed from lessons 20 to 27

src/features/intake/ and src/app/api/intake/ hold the feature; src/features/intake/triage/ holds lessons 26 and 27. The rules from lesson 22 are the table modernization/rules.md (columns ID | Rule | Citation | Side effects): R-01 to R-07 from lesson 22, R-08 and R-09 (time base and actor format) found by lesson 24 when the two writers began sharing tables, and the decisions R-10 (the schema contract, lesson 24) and R-11 (mismatch rulings, lesson 25). Unresolved questions are in modernization/unknowns.md (U-01 to U-10). The golden corpus, the recorded property set and tested-elsewhere.json are in modernization/golden/, and the differential harness from lesson 23 is tests/differential/ run by npm run test:diff, which now includes the property replay. Lesson 24’s are scripts/db/ (snapshot and verify-intact.mjs), db/schema-contract.json with src/db/contract.ts, evidence/db-intact-report.json, and the environment tests in tests/db/*.env.test.ts. Lesson 25’s are stryker.config.json, modernization/referee-paths.txt with scripts/parity/check-referee.mjs, modernization/parity/proposals/ and rulings/ with scripts/parity/check-rulings.mjs, and evidence/parity-report.json. Rules this lesson’s model behavior adds start at R-12. Lesson 20’s outputs are evidence/sbom.cdx.json and evidence/evidence-manifest.json, made and checked by scripts/sbom/finalize-sbom.mjs, verify-sbom.mjs and evidence-manifest.mjs (there is no npm run sbom; the generator is npx cyclonedx-npm). Lesson 21’s are compliance/statements/ac-12.json and ac-3.json, checked by node scripts/compliance/check-control-statements.mjs. Lesson 22’s rule check is modernization/scripts/check-rules.sh. Operator notes are docs/intake/operator-notes.md (new in this lesson). The variant files are modernization/variant.json (yours) and modernization/supported-variants.json (the registry of what each choice adds), checked by scripts/capstone/check-variant.mjs, whose tests are tests/unit/check-variant.test.ts in the runnable reference app. Rename to match what your branch has.

Plan the change with your agent

Start your agent in the project root and switch to Plan mode (Shift+Tab, /plan, or the mode picker in your desktop app). Ask it to plan the final pull-request assembly, the cold-runner verification script, the behavior traceability check, and the gated live endpoint test:

We are packaging the intake modernization as one reviewable pull request. Plan modernization/variant.json, scripts/verify-cold.sh, evidence/traceability.json with scripts/check-trace.ts, and the gated live triage test. Ask me anything unclear before writing code.

What to check in the plan before you approve

  • Variant declaration without weakening the referee: modernization/variant.json declares cloud, boundaryProfile, codeHostEdition, legacyStack, and appType, validated by scripts/capstone/check-variant.mjs against modernization/supported-variants.json without editing any referee path.
  • One cold-verification command with separate outcomes: scripts/verify-cold.sh (npm run verify:cold) unsets INTAKE_LIVE and DB_ENV, runs every non-live check in sequence (lockfile install, typecheck, unit, differential and property replay, rules check, schema-contract tests, Stryker mutation, check-referee.mjs, check-rulings.mjs, triage red-team and review tests, model-off e2e, build, SBOM regeneration and compare-sbom.mjs / verify-sbom.mjs, verify-deployed-digest.mjs, evidence-manifest.mjs verify, check-control-statements.mjs, check-trace.ts, host pipeline YAML parse, and check-variant.mjs), prints explicit NOT RUN lines for the environment tests, and outputs Local exercise and Deployment evidence on two separate lines (never a single combined verdict).
  • Mechanical end-to-end traceability: evidence/traceability.json links each shipped behavior to a rule ID in modernization/rules.md (R-12 and up for model behaviors with no legacy rule), at least one existing test file and test name, at least one hashed file in evidence/evidence-manifest.json, and a named human owner (first.last), verified by scripts/check-trace.ts.
  • Gated live endpoint test: src/features/intake/triage/triage.live.test.ts runs only when INTAKE_LIVE=1, never imports a mock provider, and passes only when triage() returns status: 'suggested' with matching deployment metadata.
  • Report failures instead of editing checks: Any failing check during assembly is reported for repair in the code or deterministic evidence generator, never silenced by loosening a check script.

If the first plan reports a single "ALL CHECKS PASSED" summary or lets the live endpoint test pass on a { status: 'manual' } fallback, correct it before approving:

Never combine Local exercise and Deployment evidence into one verdict, and make triage.live.test.ts fail if the outcome is manual instead of suggested.

The ten parts of the pull request

PartComes fromLives atA reviewer can tell it is good if
RulesLesson 22modernization/rules.mdevery rule has an id, a citation to where the legacy system or desk guide shows it, and its side effects; open questions sit in modernization/unknowns.md with an owner
Differential testsLesson 23tests/differential/ and modernization/golden/each rule has a corpus case or a named test in tested-elsewhere.json, and npm run test:diff compares old and new observations (response, intake row, audit row)
Database intact and contractLesson 24scripts/db/, db/schema-contract.json, evidence/db-intact-report.json, evidence/db-contract-report.jsonthe contract tests pass, the intact report says intact: true, the contract report lists no violations, both hashes are in the manifest, and the environment run that produced it names who ran it and against which restores
Parity benchLesson 25stryker.config.json, tests/parity/, modernization/parity/, evidence/parity-report.jsonmutation score is at its break threshold, every referee path has a code owner, and every mismatch in the report has a ruling by a named person on its exact evidence
Model-backed triageLessons 26 and 27src/features/intake/triage/the red-team and review tests pass with the mock, and the model-off path completes
SBOMLesson 20evidence/sbom.cdx.jsonregenerating it with cyclonedx-npm and finalize-sbom.mjs yields the same component identities under compare-sbom.mjs, and verify-sbom.mjs passes
Evidence manifestLesson 20evidence/evidence-manifest.jsonit names the image digest it was written for, every file has a hash, and evidence-manifest.mjs verify recomputes them
Control statementLesson 21compliance/statements/ac-12.json, ac-3.jsoncheck-control-statements.mjs passes: it says what the system supports evidence for, every pointer resolves to a file that contains its anchor, and a named human signed it
Your variantLessons 28 to 37modernization/variant.json, plus the extension checks your choices addcheck-variant.mjs accepts the file, each extension check it lists is a named step or a named environment test, and the file is a referee path so the agent cannot edit its way out of a check
Operator notesThis lessondocs/intake/operator-notes.mdsomeone on call can turn the model off, read the logs and roll back without the author

Operator notes are the one part not built earlier. They hold: how to switch the model off (set MODEL_ENABLED to anything but true and restart), what a manual outcome means and that it is not an incident, which log fields exist and which are deliberately absent (no narratives, no raw model output), how to roll back to the legacy route, who owns the model deployment, and the live-test runbook below.

Traceability: every behavior has a rule, a test, evidence and an owner

REFERENCE CONFIG evidence/traceability.json: behavior-to-rule-to-test-to-evidence-to-owner map open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
{
"behaviors": [
{
"id": "B-01",
"behavior": "A complete intake submission creates a case with the legacy routing result",
"rule": "R-04",
"tests": [{ "file": "tests/differential/intake.diff.test.ts", "name": "c01-routine-base" }],
"evidence": ["evidence/differential-report.json"],
"owner": "j.rivera"
},
{
"id": "B-02",
"behavior": "Triage suggestion is accepted only if it passes the strict output schema",
"rule": "R-12",
"tests": [{ "file": "src/features/intake/triage/triage.redteam.test.ts", "name": "malformed model JSON: fails closed to the manual path" }],
"evidence": ["evidence/redteam-log.txt"],
"owner": "a.okafor"
},
{
"id": "B-03",
"behavior": "Oversized narratives are refused before any model call",
"rule": "R-13",
"tests": [{ "file": "src/features/intake/triage/triage.redteam.test.ts", "name": "oversized input: rejected before any provider call" }],
"evidence": ["evidence/redteam-log.txt"],
"owner": "a.okafor"
},
{
"id": "B-04",
"behavior": "A caseworker decision records actor, time, decision and reason",
"rule": "R-14",
"tests": [{ "file": "src/features/intake/triage/triage-review.test.ts", "name": "a modified decision stores actor, time, decision and reason, and updates the case" }],
"evidence": ["evidence/review-test-log.txt"],
"owner": "m.chen"
},
{
"id": "B-05",
"behavior": "With the model off, a caseworker still triages",
"rule": "R-15",
"tests": [{ "file": "e2e/intake-triage-manual.spec.ts", "name": "manual triage completes with the model off" }],
"evidence": ["evidence/e2e-report.txt"],
"owner": "m.chen"
},
{
"id": "B-06",
"behavior": "No schema change merges that a frozen legacy dependent cannot survive",
"rule": "R-10",
"tests": [{ "file": "tests/db/schema-contract.test.ts", "name": "refuses every change a frozen dependent cannot survive" }],
"evidence": ["evidence/db-contract-report.json"],
"owner": "j.rivera"
},
{
"id": "B-07",
"behavior": "Every parity mismatch carries a ruling by a named person on its exact evidence",
"rule": "R-11",
"tests": [{ "file": "tests/parity/check-rulings.test.ts", "name": "a proposal alone is not a ruling" }],
"evidence": ["evidence/parity-report.json"],
"owner": "j.rivera"
}
]
}

R-01 to R-09 come from the legacy system, and R-10 and R-11 record the decisions lessons 24 and 25 made. Model behavior has no legacy rule to trace to, so you add new rules, R-12 and up, to modernization/rules.md, each citing the decision file it records under modernization/decisions/, with who made it, and each named in modernization/golden/tested-elsewhere.json with its test of record (the red-team fixtures, since there is no legacy output to compare). One row as a pattern: | R-12 | a triage suggestion is used only if it passes the strict output schema | decisions/D-03-triage-output-gate.md:4 | none |. Pointing model behavior at a legacy rule to fill the column would make the trace look complete and be false. A behavior that depends on an open question traces to its U- entry instead, and the checker below should be extended to accept that prefix. The test names in the sample are placeholders: use the exact names in your files.

REFERENCE CODE scripts/check-trace.ts: verify every behavior traces to rules.md, a test, the manifest, and a human owner open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
scripts/check-trace.ts
import { readFileSync, existsSync } from 'node:fs';
type Entry = { id: string; rule: string; tests: { file: string; name: string }[]; evidence: string[]; owner: string };
const trace = JSON.parse(readFileSync('evidence/traceability.json', 'utf8')).behaviors as Entry[];
// Lesson 20's manifest: { image: { digest }, files: { "<path>": "sha256:<hash>" } }. A file path is the evidence id.
const manifestIds = new Set(
Object.keys(JSON.parse(readFileSync('evidence/evidence-manifest.json', 'utf8')).files as Record<string, string>),
);
const spec = readFileSync('modernization/rules.md', 'utf8');
const problems: string[] = [];
for (const b of trace) {
if (!new RegExp(`\\b${b.rule}\\b`).test(spec)) problems.push(`${b.id}: rule ${b.rule} is not in modernization/rules.md`);
if (b.tests.length === 0) problems.push(`${b.id}: no test`);
for (const t of b.tests) {
if (!existsSync(t.file)) problems.push(`${b.id}: test file missing ${t.file}`);
else if (!readFileSync(t.file, 'utf8').includes(t.name)) problems.push(`${b.id}: no test named "${t.name}" in ${t.file}`);
}
if (b.evidence.length === 0) problems.push(`${b.id}: no evidence item`);
for (const e of b.evidence) if (!manifestIds.has(e)) problems.push(`${b.id}: evidence file ${e} not in the manifest`);
if (!/^[a-z]+\.[a-z]+$/.test(b.owner)) problems.push(`${b.id}: owner must be a named person (first.last)`);
}
if (problems.length) { console.error(problems.join('\n')); process.exit(1); }
console.log(`traceability: ${trace.length} behaviors, all traced`);

The checker finds a test name as text in the file, so a name in a comment or a skipped test also passes it: it proves the trace points at something that exists, not that the test ran. The run itself is the unit and differential lines above, and the environment results in the pull request.

Hash checking is lesson 20’s scripts/sbom/evidence-manifest.mjs verify evidence/evidence-manifest.json <image-digest>. It fails if the manifest was written for a different image or if any listed file has changed since. It records no owner per file; the owner of each piece of evidence is the owner on the trace entry that cites it.

Evidence that is itself generated by a test run (a report, a log) must be generated and hashed by the same command that commits it; otherwise the cold runner’s fresh run will legitimately differ from the committed bytes. Keep committed evidence to artifacts that are deterministic, and let the cold run produce its own logs under build/verify/.

Declare your variant

The capstone is one fixed exercise, the intake route moved from classic ASP and T-SQL, run locally. That baseline always runs in full. Your situation adds to it through modernization/variant.json, and nothing in that file can take a baseline check away (reference baseline file from gov-dev-reference-app):

{
"cloud": "local",
"boundaryProfile": "none",
"codeHostEdition": "github-cloud",
"legacyStack": "classic-asp-sqlserver",
"appType": "intake"
}

Save it as modernization/variant.json. Each field has a short list of values, and each value points at the lesson that teaches what it adds:

FieldValuesTaught in
cloudlocal, azure, aws, google34, 35, 36
boundaryProfilenone, fedramp-moderate, fedramp-high, il4, il5 (the authorization level the cloud must sit inside: FedRAMP is the federal cloud authorization program, IL4 and IL5 are the Defense Department’s impact levels)the platform lesson for your cloud
codeHostEditiongithub-cloud, github-enterprise-server, gitlab-saas, gitlab-self-managed, gitlab-dedicated-government, azure-devops-services, azure-devops-server, bitbucket-cloud37
legacyStackclassic-asp-sqlserver (the baseline), oracle-plsql, db2-cobol, java-ee, dotnet-framework, access22, 28, 29
appTypeintake (the baseline), batch, workflow, portal, reports30, 31, 32, 33

The registry in modernization/supported-variants.json says what each value adds, in two lists per value: local checks that run on a cold clone, and deployment checks that need a live environment and a named person (reference excerpt from gov-dev-reference-app):

// modernization/supported-variants.json (excerpt)
"baseline": {
"local": ["typecheck", "unit", "contract", "differential", "property-replay", "mutation", "referee", "rulings", "traceability", "evidence-hashes", "db-contract-rules"],
"deployment": ["live-endpoint", "db-intact", "db-contract", "parity-live", "db-env-tests", "triage-grants"]
},
// ...
"azure-devops-services": { "local": ["pipeline-yaml:azure-devops"], "deployment": ["required-reviewer-policies:azure-devops", "oidc-federation:azure-devops"] },
// ...
"batch": { "local": ["batch-keyed-diff", "batch-rerun-idempotent"], "deployment": ["batch-run-window"] },
// ...
{ "when": { "cloud": "local" }, "forbid": { "boundaryProfile": ["fedramp-moderate", "fedramp-high", "il4", "il5"] }, "why": "a laptop has no authorization boundary" },

The script reads both files, refuses anything the registry does not allow, and reports two outcomes:

REFERENCE CODE scripts/capstone/check-variant.mjs: validate variant.json and report separate local and deployment outcomes open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// scripts/capstone/check-variant.mjs — the capstone's variant (lesson 38).
// modernization/variant.json declares what this rebuild is; modernization/supported-variants.json
// (a referee file) says what each choice ADDS. The baseline always runs: a variant can add
// obligations and never remove one. Two outcomes, reported separately:
// local every local check passed (ci-out/checks/<id>.pass exists)
// deployment every deployment check has evidence (evidence/deployment/<id>.json, with
// "result": "pass" and a named "by"); a NOT RUN line never counts here.
// Usage: node scripts/capstone/check-variant.mjs [repo-root] -> exit 0 only if the variant is valid.
import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';
const root = process.argv[2] ?? '.';
const read = (p) => JSON.parse(readFileSync(join(root, p), 'utf8'));
const registry = read('modernization/supported-variants.json');
const variant = read('modernization/variant.json');
const problems = [];
const FIELDS = Object.keys(registry.fields);
for (const k of Object.keys(variant)) if (!FIELDS.includes(k)) problems.push(`variant.json: unknown field "${k}"`);
for (const k of FIELDS) {
if (!(k in variant)) problems.push(`variant.json: missing "${k}" (one of: ${Object.keys(registry.fields[k]).join(', ')})`);
else if (typeof variant[k] !== 'string' || !Object.hasOwn(registry.fields[k], variant[k])) problems.push(`variant.json: ${k} "${variant[k]}" is not supported (one of: ${Object.keys(registry.fields[k]).join(', ')})`);
}
for (const rule of registry.incompatible) {
const applies = Object.entries(rule.when).every(([k, v]) => variant[k] === v);
if (applies) for (const [k, bad] of Object.entries(rule.forbid)) if (bad.includes(variant[k])) problems.push(`variant.json: ${k} "${variant[k]}" with ${JSON.stringify(rule.when)}: ${rule.why}`);
}
if (problems.length) { console.error(problems.join('\n')); process.exit(1); }
const add = (kind) => [...new Set([...registry.baseline[kind], ...FIELDS.flatMap((k) => registry.fields[k][variant[k]][kind])])];
const local = add('local');
const deployment = add('deployment');
const localDone = local.filter((id) => existsSync(join(root, `ci-out/checks/${id}.pass`)));
const deployDone = deployment.filter((id) => {
const f = join(root, `evidence/deployment/${id.replace(/[:/]/g, '_')}.json`);
if (!existsSync(f)) return false;
try { const e = JSON.parse(readFileSync(f, 'utf8')); return e.result === 'pass' && typeof e.by === 'string' && e.by.length > 0; }
catch { return false; }
});
for (const id of local) console.log(`${localDone.includes(id) ? 'PASS ' : 'MISSING'} local ${id}`);
for (const id of deployment) console.log(`${deployDone.includes(id) ? 'PASS ' : 'NOT RUN'} deployment ${id}`);
console.log(`Local exercise: ${localDone.length === local.length ? 'COMPLETE' : 'INCOMPLETE'} (${localDone.length}/${local.length})`);
console.log(`Deployment evidence: ${deployDone.length === deployment.length ? 'COMPLETE' : 'INCOMPLETE'} (${deployDone.length}/${deployment.length})`);

Its tests pin the rules that matter. The baseline lists every baseline check and is incomplete until each passes. An extension adds obligations and never removes a baseline one. Complete local checks do not make deployment complete, and a NOT RUN never counts. Deployment evidence counts only with result: pass and a named person. An unsupported value, a missing field, an unknown field and an incompatible pair are all refused:

REFERENCE TEST tests/unit/check-variant.test.ts (excerpt): unit tests for variant obligations and separate outcome reporting open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// tests/unit/check-variant.test.ts (excerpt)
const baseline = { cloud: 'local', boundaryProfile: 'none', codeHostEdition: 'github-cloud', legacyStack: 'classic-asp-sqlserver', appType: 'intake' };
describe('capstone variant', () => {
it('the baseline variant lists every baseline check and is incomplete until each passes', () => {
const r = run(baseline);
expect(r.code).toBe(0);
expect(r.out).toMatch(/MISSING local mutation/);
expect(r.out).toMatch(/Local exercise: INCOMPLETE \(0\/12\)/);
});
it('an extension adds obligations and never removes a baseline one', () => {
const r = run({ ...baseline, appType: 'batch' });
expect(r.out).toMatch(/local batch-rerun-idempotent/);
expect(r.out).toMatch(/local mutation/);
});
it('local checks complete does not make deployment complete; NOT RUN never counts', () => {
const checks = Object.fromEntries(['typecheck', 'unit', 'contract', 'differential', 'property-replay', 'mutation', 'referee', 'rulings', 'traceability', 'evidence-hashes', 'db-contract-rules', 'pipeline-yaml:github'].map((id) => [`ci-out/checks/${id}.pass`, '']));
const r = run(baseline, checks);
expect(r.out).toMatch(/Local exercise: COMPLETE/);
expect(r.out).toMatch(/Deployment evidence: INCOMPLETE \(0\/8\)/);
});
it('deployment evidence counts only with result pass and a named person', () => {
const r = run({ ...baseline, cloud: 'azure', boundaryProfile: 'fedramp-high' }, {
'evidence/deployment/signin_azure.json': JSON.stringify({ result: 'pass', by: 'j.rivera' }),
'evidence/deployment/deploy_azure.json': JSON.stringify({ result: 'pass' }),
});
expect(r.out).toMatch(/PASS deployment signin:azure/);
expect(r.out).toMatch(/NOT RUN deployment deploy:azure/);
});
it('refuses an unsupported value, a missing field, an unknown field, and an incompatible pair', () => {
expect(run({ ...baseline, appType: 'mainframe-ui' }).out).toMatch(/appType "mainframe-ui" is not supported/);
expect(run({ ...baseline, appType: 'toString' }).out).toMatch(/appType "toString" is not supported/);
const { appType, ...noApp } = baseline;
expect(run(noApp).out).toMatch(/missing "appType"/);
expect(run({ ...baseline, skipChecks: ['mutation'] }).out).toMatch(/unknown field "skipChecks"/);
const r = run({ ...baseline, boundaryProfile: 'il4' });
expect(r.code).toBe(1); expect(r.out).toMatch(/a laptop has no authorization boundary/);
});
});

Run in the reference app, cd gov-dev-reference-app && npx vitest run tests/unit/check-variant.test.ts, the real output is:

RUN v5.1.0 <reference app>
Test Files 1 passed (1)
Tests 5 passed (5)
Start at 15:06:05
Duration 171ms (tests 92%, import 5%, transform 2%, worker 2%)

Two outcomes are reported everywhere this capstone reports a result, and they are never combined. Local exercise: COMPLETE means every local check passed on a cold clone. Deployment evidence: COMPLETE means every deployment check has a file at evidence/deployment/<id>.json (the id with : and / replaced by _, so signin:azure is signin_azure.json) whose result is pass and whose by names a person, such as { "result": "pass", "by": "j.rivera" }. A file with no by is counted as NOT RUN, and so is a missing file. A NOT RUN line never counts toward deployment, however many local checks are green.

The two files are referee paths. In gov-dev-reference-app, they are included in lesson 25’s modernization/referee-paths.txt and the minimum list inside check-referee.mjs so that shortening the file fails the build (reference excerpt):

# modernization/referee-paths.txt (additions)
modernization/variant.json
modernization/supported-variants.json

Changing the variant is therefore a referee-path change: check-referee.mjs requires a code owner for it, and the pull-request check from lesson 25 fails any pull request the agent opens that touches it. An agent that finds a batch check failing cannot edit appType back to intake to make the failure go away.

The one command

REFERENCE CODE scripts/verify-cold.sh: single cold-runner verification script open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
#!/usr/bin/env bash
# scripts/verify-cold.sh — run via: npm run verify:cold
set -u
cd "$(dirname "$0")/.."
: "${IMAGE_DIGEST:?set IMAGE_DIGEST to the sha256 digest of the image under review (it is in the PR description)}"
: "${DEPLOYED_IMAGE_REF:?set DEPLOYED_IMAGE_REF to the image reference the deployed app reports, e.g. registry/app@sha256:...}"
mkdir -p build/verify ci-out/checks
rm -f ci-out/checks/*.pass # a cold run earns every marker itself
unset INTAKE_LIVE DB_ENV # a cold run never picks up environment tests from the caller's shell
# The declared code host picks lesson 25's referee adapter: github-cloud -> github, azure-devops-services -> azure-devops, and so on.
REPO_HOST=$(node -e 'const h=require("./modernization/variant.json").codeHostEdition;console.log(h.startsWith("azure-devops")?"azure-devops":h.split("-")[0])')
export REPO_HOST
results=()
run() {
local name="$1"; shift
if "$@" > "build/verify/${name}.log" 2>&1; then results+=("PASS ${name}"); : > "ci-out/checks/${name}.pass" # the marker check-variant.mjs reads
else results+=("FAIL ${name} (see build/verify/${name}.log)"); fi
}
run install npm ci
run typecheck npx tsc --noEmit
run unit npx vitest run # config excludes *.live.test.ts and *.env.test.ts
run differential npm run test:diff
run property-replay npx vitest run tests/parity/property-replay.test.ts # lesson 25's recorded set, its own check
run contract npx vitest run tests/db/schema-contract.test.ts
run rules bash modernization/scripts/check-rules.sh
run db-contract-rules npx vitest run tests/db # contractViolations() on fixture snapshots
run mutation npx stryker run # exits 1 below thresholds.break
run referee node scripts/parity/check-referee.mjs # REPO_HOST, set above, picks the adapter
run rulings node scripts/parity/check-rulings.mjs evidence/parity-report.json
run redteam-mock npx vitest run src/features/intake/triage/triage.redteam.test.ts
run review-record npx vitest run src/features/intake/triage/triage-review.test.ts
run e2e-model-off env MODEL_ENABLED=false node scripts/e2e.mjs
run build npm run build
run sbom-regenerate bash -c 'npx cyclonedx-npm --omit dev --mc-type application --of JSON --sv 1.6 -o build/verify/sbom.cdx.json && node scripts/sbom/finalize-sbom.mjs build/verify/sbom.cdx.json "intake build pipeline"'
run sbom-matches node scripts/sbom/compare-sbom.mjs build/verify/sbom.cdx.json evidence/sbom.cdx.json
run sbom-minimum node scripts/sbom/verify-sbom.mjs evidence/sbom.cdx.json --pre-sign # signature is checked in the pipeline
run deployed-digest node scripts/verify-deployed-digest.mjs "$IMAGE_DIGEST" "$DEPLOYED_IMAGE_REF"
run evidence-hashes node scripts/sbom/evidence-manifest.mjs verify evidence/evidence-manifest.json "$IMAGE_DIGEST"
run control-statements node scripts/compliance/check-control-statements.mjs
run traceability npx tsx scripts/check-trace.ts
# Extension steps for your variant. Each is added with this same run helper, named exactly as the registry
# names it (ids contain ':'; the marker is ci-out/checks/<id>.pass), so each earns its own marker.
# The declared host's pipeline file must parse; unknown tags (GitLab's !reference) are ignored.
shopt -s nullglob # a pattern that matches nothing leaves the list empty, and the step fails below
case "$REPO_HOST" in
github) PIPELINE_FILES=(.github/workflows/*.yml .github/workflows/*.yaml) ;;
gitlab) PIPELINE_FILES=(.gitlab-ci.yml) ;;
bitbucket) PIPELINE_FILES=(bitbucket-pipelines.yml) ;;
azure-devops) PIPELINE_FILES=(azure-pipelines.yml) ;;
esac
run "pipeline-yaml:${REPO_HOST}" python3 -c 'import sys, yaml
class L(yaml.SafeLoader): pass
L.add_multi_constructor("!", lambda loader, suffix, node: None)
assert sys.argv[1:], "no pipeline file found for this host"
for f in sys.argv[1:]: yaml.load(open(f), Loader=L)' "${PIPELINE_FILES[@]}"
# Other extension ids (collector-validate, inventory, referee-approval-check:bitbucket) are added the same way.
run variant node scripts/capstone/check-variant.mjs # last: it reads the markers above
printf '%s\n' "${results[@]}"
echo "NOT RUN live-endpoint (environment test; needs your tenant: npm run verify:live)"
echo "NOT RUN db-intact (environment test; two restores inside the boundary: see docs/intake/operator-notes.md)"
echo "NOT RUN db-contract (environment test; migrations on a disposable restore: npm run db:contract)"
echo "NOT RUN db-env-tests (environment test; time base and actor, lesson 24: DB_ENV=1 npx vitest run tests/db)"
echo "NOT RUN triage-grants (environment test; grants and race, lesson 27: INTAKE_LIVE=1 npx vitest run src/features/intake/triage/triage-grants.live.test.ts src/features/intake/triage/triage-race.live.test.ts)"
echo "NOT RUN parity-live (environment test; the nightly bench, lesson 25)"
# The six lines above are the baseline deployment list. build/verify/variant.log lists every check your variant adds.
grep -E '^(Local exercise|Deployment evidence):' build/verify/variant.log # two outcomes, never combined
! printf '%s\n' "${results[@]}" | grep -q '^FAIL' && grep -q '^Local exercise: *COMPLETE' build/verify/variant.log

The script exits 0 only when no step failed and the local exercise is complete. It never exits on deployment evidence, which the cold runner cannot produce; that line stays INCOMPLETE until named people attach their runs. The SBOM comparison is a Node script because a bash process substitution (diff <(...) <(...)) does not reliably fail the comparison when the command inside it fails: if the generator crashes, both sides can come out empty and diff reports no difference.

REFERENCE CODE scripts/sbom/compare-sbom.mjs: multiset comparison of regenerated and committed SBOM component identities open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// scripts/sbom/compare-sbom.mjs — usage: node compare-sbom.mjs <regenerated.json> <committed.json>
import { readFileSync } from "node:fs";
function identities(path) {
let bom;
try { bom = JSON.parse(readFileSync(path, "utf8")); }
catch (e) { throw new Error(`${path}: cannot read as JSON (${e.name})`); }
if (bom?.bomFormat !== "CycloneDX") throw new Error(`${path}: bomFormat is not CycloneDX`);
if (!Array.isArray(bom.components) || bom.components.length === 0) throw new Error(`${path}: no components`);
const bad = bom.components.filter((c) => typeof c?.purl !== "string" || c.purl.trim() === "");
if (bad.length) throw new Error(`${path}: ${bad.length} component(s) with no purl, first: ${bad[0]?.name ?? "(unnamed)"}`);
return bom.components.map((c) => c.purl.trim()).sort(); // a sorted list, so a duplicate counts
}
try {
const [a, b] = [identities(process.argv[2]), identities(process.argv[3])]; // both sorted
// Element by element, not "same set and same length": [p,p,q] and [p,q,q] have the same set and the same length.
const same = a.length === b.length && a.every((x, i) => x === b[i]);
if (!same) {
const count = (list) => list.reduce((m, x) => m.set(x, (m.get(x) ?? 0) + 1), new Map());
const ca = count(a), cb = count(b);
console.error(`SBOM identities differ (${a.length} vs ${b.length})`);
for (const x of new Set([...ca.keys(), ...cb.keys()])) {
const [na, nb] = [ca.get(x) ?? 0, cb.get(x) ?? 0];
if (na !== nb) console.error(` ${x}: ${na}x in regenerated, ${nb}x in committed`);
}
process.exit(1);
}
console.log(`sbom: ${a.length} components, identical identities`);
} catch (e) {
console.error(`sbom: ${e.message}`);
process.exit(1);
}

Any unreadable file, empty component list or component with no identifier exits non-zero before a comparison is even attempted. The comparison itself walks the two sorted lists position by position, so a duplicated identity on one side and a missing one on the other cannot cancel out. The identifier is the package URL (purl); if your generator keys components another way, change identities() and keep the rule that an empty identifier is an error.

Lesson 19 exports the imageDigest it built and deploys by that digest, not by a tag. The cold run takes that digest as IMAGE_DIGEST and the reference the deployed app reports as DEPLOYED_IMAGE_REF, and compares them with the reference scripts/verify-deployed-digest.mjs script from gov-dev-reference-app:

// scripts/verify-deployed-digest.mjs — usage: node verify-deployed-digest.mjs <imageDigest> <deployedRef>
const [digest, ref] = process.argv.slice(2);
const DIGEST = /^sha256:[0-9a-f]{64}$/;
if (!DIGEST.test(digest ?? "")) { console.error("IMAGE_DIGEST is not a sha256 digest"); process.exit(1); }
const at = (ref ?? "").lastIndexOf("@");
if (at < 1) { console.error("deployed reference is not digest-qualified (no @sha256:...): deployed by tag?"); process.exit(1); }
if (ref.slice(at + 1) !== digest) { console.error(`deployed digest ${ref.slice(at + 1)} != reviewed digest ${digest}`); process.exit(1); }
console.log(`deployed reference carries the reviewed digest ${digest}`);

How you read the deployed reference back (the platform’s app configuration, the release record) is the part your environment decides; the point is that the value comes from the deployed side, not from the pipeline’s own variable.

Two choices in this script are deliberate. The step names that use a mock say so (redteam-mock), so a green line cannot be read as proof about a real endpoint. And the NOT RUN lines always print, so every environment test is a fact in each cold run’s output rather than something nobody mentioned. The cold run still checks their committed results: the intact report and the parity report are hashed in the manifest, rulings reads the parity report, and evidence-hashes fails if either file changed after it was hashed. What it cannot check is that the environment run happened recently and against the right restores; the pull request says who ran each one, when, and against what. The e2e step runs against lesson 13’s local test stack and its seeded test database; if your stack has no local database, list that check as an environment test too. It needs the browser binaries installed once; put that in the README’s setup section so the runner’s one command is really one command.

The live environment test is separate and named

REFERENCE TEST src/features/intake/triage/triage.live.test.ts: gated live endpoint environment test open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
src/features/intake/triage/triage.live.test.ts
import { describe, it, expect } from 'vitest';
import { triage } from './service';
import { getProvider } from './get-provider';
import { TriageSuggestionSchema } from './schema';
import { PROMPT_HASH } from './prompt';
const live = process.env.INTAKE_LIVE === '1';
describe.skipIf(!live)('LIVE endpoint (environment test; never run with a mock)', () => {
it('returns a schema-valid suggestion from the configured deployment, not a fallback', async () => {
const lines: Record<string, unknown>[] = [];
const out = await triage({ caseId: 'LIVE-1', narrative: 'Synthetic test narrative: request for a records copy.' },
{ provider: getProvider, log: (l) => lines.push(l), now: () => new Date() }); // the factory, uncalled: a config fault surfaces as provider_misconfigured
console.log(lines.map((l) => JSON.stringify(l)).join('\n'));
// MODEL_ENABLED unset gives manual / model_disabled; a bad config gives manual / provider_misconfigured.
// "manual" is how triage() hides a broken endpoint from the caseworker, so it cannot pass here.
expect(out.status, JSON.stringify(lines)).toBe('suggested');
if (out.status !== 'suggested') return;
expect(TriageSuggestionSchema.safeParse(out.suggestion).success).toBe(true);
expect(out.meta.deployment).toBe(process.env.MODEL_DEPLOYMENT);
expect(out.meta.modelVersion).not.toBe('unrecorded');
expect(out.meta.promptHash).toBe(PROMPT_HASH);
expect(out.ai_drafted).toBe(true);
expect(lines).toHaveLength(1);
expect(lines[0].outcome).toBe('suggested');
expect(['provider_unavailable', 'provider_misconfigured']).not.toContain(lines[0].reason);
});
});

And the reference package.json script entries from gov-dev-reference-app:

{ "scripts": { "verify:cold": "bash scripts/verify-cold.sh", "verify:live": "INTAKE_LIVE=1 vitest run src/features/intake/triage/triage.live.test.ts" } }

A file excluded by the Vitest config stays excluded even when you name it on the command line, so the exclusion in the reference vitest.config.ts from gov-dev-reference-app depends on the same variable the live test reads:

// vitest.config.ts (excerpt)
import { defineConfig, configDefaults } from 'vitest/config';
export default defineConfig({
test: {
exclude: [
...configDefaults.exclude,
'e2e/**', // Playwright specs, run by scripts/e2e.mjs
...(process.env.INTAKE_LIVE === '1' ? [] : ['**/*.live.test.ts']),
...(process.env.DB_ENV === '1' ? [] : ['**/*.env.test.ts']),
],
},
});

The test accepts one outcome, suggested. Lesson 26’s triage code turns a provider failure into manual on purpose, so a test that also accepted manual would pass against a broken endpoint, a revoked role assignment or a wrong model id.

Live testing is run by a named person from inside the authorization boundary, with a synthetic narrative only, and its output is attached to the pull request as a separate section: who ran it, when, against which environment name (never a secret or full endpoint), and the log lines. Whether the endpoint authenticates with a managed identity, answers within the timeout, and returns the shape the adapter expects are exactly the claims the mocked suite cannot make, which lesson 26 left to this test.

SESSION Cold runner: a fresh session with only the repo and the README
  1. Typed in the terminal: claude
  2. Mode:Plan mode
  3. You type in the session: Read README.md and plan how to run the single cold verification command it documents without fixing any failures.
  4. The tool replies: README.md documents a one-time Playwright browser install followed by npm run verify:cold with IMAGE_DIGEST and DEPLOYED_IMAGE_REF set. Should I switch out of Plan mode, run that command as written, and report PASS or FAIL for each step plus any documentation questions?
  5. You type in the session: Yes. Run the single verification command it documents, exactly as written. Do not fix anything. Under each result write PASS or FAIL, and list any question you would have asked me.
  6. The tool replies: README.md says: install the pinned runtime (see .nvmrc), run npx playwright install chromium once, then npm run verify:cold. Running the setup lines, then npm run verify:cold. PASS install PASS typecheck PASS unit PASS differential PASS property-replay PASS contract PASS rules PASS db-contract-rules PASS mutation PASS referee PASS rulings PASS redteam-mock PASS review-record PASS e2e-model-off PASS build PASS sbom-regenerate PASS sbom-matches PASS sbom-minimum PASS deployed-digest FAIL evidence-hashes (see build/verify/evidence-hashes.log) PASS control-statements PASS traceability PASS pipeline-yaml:github PASS variant NOT RUN live-endpoint (environment test; needs your tenant: npm run verify:live) NOT RUN db-intact (environment test; two restores inside the boundary: see docs/intake/operator-notes.md) NOT RUN db-contract (environment test; migrations on a disposable restore: npm run db:contract) NOT RUN db-env-tests (environment test; time base and actor, lesson 24: DB_ENV=1 npx vitest run tests/db) NOT RUN triage-grants (environment test; grants and race, lesson 27: INTAKE_LIVE=1 npx vitest run src/features/intake/triage/triage-grants.live.test.ts src/features/intake/triage/triage-race.live.test.ts) NOT RUN parity-live (environment test; the nightly bench, lesson 25) Local exercise: INCOMPLETE (11/12) Deployment evidence: INCOMPLETE (0/8) Failure detail: evidence: FAIL evidence/differential-report.json changed since the manifest was written Questions: none for the steps that ran. One documentation gap: the README does not say which runtime version the build was verified on, I used the one in .nvmrc.
  7. You type in the session: Do not fix it. Tell me why that hash could differ on a clean checkout, and which of my own checks should have caught it.
  8. The tool replies: Most likely cause, from reading the log and the manifest: evidence/differential-report.json contains a generated-at timestamp, so its bytes differ on every run, but the manifest pins one hash for it. The check that caught it is evidence-manifest.mjs verify, which is working as designed: the committed file and its recorded hash disagree after a fresh run regenerated it. What should have caught it earlier: nothing in the builder's own run, because the builder ran the same command in the same checkout, where the file and the manifest were produced together. That is the point of a cold run. I have not changed anything. The repair is yours: either strip the timestamp from the committed report or move the report out of the manifest and into the per-run logs, then re-run.
  9. You type in the session: I removed the timestamp and re-hashed. Run only the failed check again, then give me the gap log entry.
  10. The tool replies: PASS evidence-hashes PASS variant Local exercise: COMPLETE (12/12) Deployment evidence: INCOMPLETE (0/8) Gap log: 1. Failed check: evidence-hashes. Cause: a timestamp inside a hashed artifact. Changed: removed the timestamp field from the differential report writer and updated the manifest. Check to add next time: a second consecutive run of verify:cold in the same checkout must produce identical hashes. 2. Runner question as documentation defect: the README lacks the verified runtime version. Fix: add it beside the setup lines. The six environment tests were NOT RUN by this session, as designed. Each remains its owner's, with the date and environment name to be recorded in the PR.

Illustrative session; exact output varies by tool and agency configuration.

What the cold runner sees

The runner has a fresh session or a fresh checkout, the README, and nothing else. Their walk-through is deliberately short:

  1. Read README.md: the runtime pin, the one-time browser install, how to get IMAGE_DIGEST (it is in the pull request description) and DEPLOYED_IMAGE_REF (the reference the deployed app reports), and the line npm run verify:cold.
  2. Run it. Read one line of PASS or FAIL per check and one NOT RUN line per environment test.
  3. Open the log of any failure under build/verify/.
  4. Open evidence/traceability.json, pick any behavior, and follow it: the rule in modernization/rules.md, the named test, the evidence file in the manifest, and the owner’s name.
  5. Write the result under each check and any question in a Questions section. They fix nothing.

A question the runner has to ask is a defect in the README, not a failure of the grade; a check that cannot run without the answer is a failure.

The pull request description

Use this pull request description structure so reviewers see the cold verification command, the separate environment test records, and the traceability file:

REFERENCE CONFIG Pull request description template for the intake modernization capstone open reference hide reference Reference file pattern for your workspace agent; open to inspect or compare. You do not type or paste this by hand.
## What this changes
Moves the intake route to the intake feature on the reference app; adds a model-backed
triage suggestion a caseworker can accept, modify or reject.
## Verify (non-live)
Image digest under review: <sha256:...> # the cold runner exports it as IMAGE_DIGEST
Deployed reference: <registry/app@sha256:...> # read back from the deployed app; exported as DEPLOYED_IMAGE_REF
npm run verify:cold # expected: all PASS, "Local exercise: COMPLETE", the six baseline environment tests NOT RUN, "Deployment evidence: INCOMPLETE" until each is attached
Variant: <contents of modernization/variant.json>
## Environment tests — each separate, each with: run by <name>, on <date>, against <environment or restore names>
- live endpoint (npm run verify:live): output attached. Status: <run / not yet run>
- database intact (verify-intact.mjs): evidence/db-intact-report.json. Status: <run / not yet run>
- database contract on a restore (npm run db:contract): evidence/db-contract-report.json. Status: <run / not yet run>
- database time base and actor (DB_ENV=1 npx vitest run tests/db): output attached. Status: <run / not yet run>
- triage grants and race (lesson 27's two .live.test.ts files): output attached. Status: <run / not yet run>
- the checks your variant adds (build/verify/variant.log): one line each, as evidence/deployment/<id>.json with result pass and a named person
- referee on Azure DevOps (if that is your host): who compared the required-reviewer policies with referee-paths.txt, and when
- live parity bench (last nightly run): evidence/parity-report.json, seed <n>, snapshot <id>. Status: <run / not yet run>
## Trace
evidence/traceability.json — <n> behaviors, each with rule, test, evidence, owner.
## Not covered
<what the mock cannot show; which behaviors differ from the legacy route and why (any rule in `modernization/rules.md` whose side effects changed, with the reason)>

The reviewer’s checklist

  • The cold runner ran the one command and reported every non-live step PASS, with zero blocking questions.
  • The output reports “Local exercise” and “Deployment evidence” as two separate lines, and the PR never says one verdict for both.
  • modernization/variant.json is accepted by check-variant.mjs, matches the system the PR actually changes, and both variant files are referee paths owned by a team.
  • Every evidence/deployment/*.json the PR cites has "result": "pass" and a named by; a file with no by, or a NOT RUN line, is counted as not done.
  • The output lists each environment test as NOT RUN, and the PR carries an environment-test section with a named runner, a date and the environment or restores for each, or says plainly that one has not been run.
  • Branch protection really requires code-owner review, in the setting lesson 37 names for your host (GitHub “Require review from Code Owners”, GitLab code owner approval on the protected branch, the Azure DevOps required-reviewer policy with path filters, or the compensating approval check on Bitbucket): check-referee.mjs, run with the REPO_HOST for your declared host, proves the owners are named, not that the setting is on.
  • The parity report compared the whole corpus and the recorded property set, and every mismatch left in it is an approved change naming its rule, ruled by someone other than its proposer. Open one ruling file’s history: the person in by is the person who approved that pull request.
  • The intact report says intact: true, and the schema contract’s frozen list matches the dependents in modernization/db-dependents.md.
  • Open any three behaviors in the trace and follow each to a rule, a test that exists, an evidence item whose hash recomputes, and a named person.
  • Each rule whose behavior changed from the legacy system has a reason and an approver, and each open U- entry has an owner.
  • No test file under the triage folder that runs in verify:cold imports a live provider; no .live.test.ts file imports a mock.
  • Logs contain no narratives and no raw model output (the red-team test asserts this).
  • The operator notes let someone who was not on the team turn the model off and roll back.
  • check-rules.sh passes, so every rule in modernization/rules.md cites something that resolves.
  • Each control statement in compliance/statements/ passes check-control-statements.mjs: it uses “supports evidence for”, every pointer resolves, and a named person signed it.
ℹStatus checked 2026-11-04

Availability, authorization, agency approval, region support, and price are five separate facts. This lesson states none of them. Whether your agency asks suppliers for an SBOM is the agency’s choice: OMB M-26-05 (23 January 2026, https://www.whitehouse.gov/wp-content/uploads/2026/01/M-26-05-Adopting-a-Risk-based-Approach-to-Software-and-Hardware-Security.pdf) rescinded the earlier memos that standardized it and says agencies may adopt contract terms that require a current SBOM on request, while agencies must still maintain a complete software and hardware inventory. Build the SBOM as engineering practice either way. Recheck the cited source and your agency’s authorization boundary before you rely on any of it.

Prove it

The capstone’s proof is the cold run, plus planted faults that the command must catch. Plant each on a scratch branch, run npm run verify:cold, and confirm the named step fails.

PlantExpected failing step
Change one expected value in a differential test so it disagrees with the legacy resultdifferential
Edit a committed evidence file by one characterevidence-hashes
Delete the owner from one trace entry, or point its test name at one that does not existtraceability
Remove one component from the committed SBOM, or blank one component’s purlsbom-matches
Regenerate against a committed SBOM whose purls are [p, p, q] when the regenerated ones are [p, q, q] (same set, same length; see the test below)sbom-matches
Set DEPLOYED_IMAGE_REF to a tag (registry/app:latest) or to a different digestdeployed-digest
Add a mismatch to evidence/parity-report.json with no ruling, or set its compared count to 0rulings (and evidence-hashes, since the report is in the manifest)
Delete if (diff > 0.5) return floor + 1; from score.tsdifferential (the property replay) and mutation
Add // Stryker disable next-line all above any line in src/features/intake/referee
Add a column to the fixture snapshot of a frozen table in tests/dbdb-contract-rules
After building a batch job, edit variant.json back to appType: "intake" so the batch checks disappearreferee and code-owner review refuse it: the variant files are referee paths, so the change cannot merge without a team owner. Run the check on the edited file and the batch checks are simply no longer listed, which is why the file, not the script, has to be protected
Write evidence/deployment/signin_azure.json with "result": "pass" and no bycounted NOT RUN by check-variant.mjs; “Deployment evidence” stays INCOMPLETE

The multiset case is the one a set comparison misses, so test it directly with two hand-written lists:

Terminal window
T=$(mktemp -d)
mk() { node -e 'const [f,...p]=process.argv.slice(1);require("fs").writeFileSync(f,JSON.stringify({bomFormat:"CycloneDX",components:p.map(x=>({name:x,purl:"pkg:npm/"+x+"@1.0.0"}))}))' "$@"; }
mk "$T/a.json" p p q # regenerated: [p, p, q]
mk "$T/b.json" p q q # committed: [p, q, q]
mk "$T/c.json" p p q # identical to a
! node scripts/sbom/compare-sbom.mjs "$T/a.json" "$T/b.json" # expect exit 1, naming p and q
node scripts/sbom/compare-sbom.mjs "$T/a.json" "$T/c.json" # expect exit 0

Pass: the first comparison exits 1 and prints pkg:npm/p@1.0.0: 2x in regenerated, 1x in committed; the second exits 0. Revert compare-sbom.mjs to the set-plus-length comparison and the first line must flip to a pass, which is the fault this test exists to catch.

Then confirm the quiet failures the script exists to prevent. Run INTAKE_LIVE=1 npm run verify:live with MODEL_ENABLED unset: it must fail because the status is manual and the log reason is model_disabled, not pass with a mock. Point MODEL_DEPLOYMENT at a model id that does not exist: it must fail because the status is manual, and the log must show provider_misconfigured, not suggested.

KNOWLEDGE CHECK

The cold run is all PASS and the last line says the live endpoint check was NOT RUN. What does the reviewer conclude?

KNOWLEDGE CHECK

A model-backed behavior (the output schema gate) has no legacy rule. How should the trace record it?

Practice status

PracticeStatusAlso called
One command reproduces every non-live checkstrong optionalreproducible verification, single entry point
A cold runner who did not build it runs the commandreference-shopindependent verification run, fresh-eyes check
Live endpoint testing kept separate and namedcommon baselineenvironment test, integration test against real service
Behavior to rule to test to evidence to owner tracestrong optionalrequirements traceability matrix
Hash-pinned evidence manifeststrong optionalevidence index, artifact manifest
SBOM committed and regenerated for comparisonemergingsoftware bill of materials, component inventory
Deployed image reference checked against the reviewed digeststrong optionaldeploy-by-digest verification, image pinning
Operator notes written for someone not on the teamcommon baselinerunbook, handoff notes
Environment tests listed as NOT RUN, with their results hashed and attributed in the PRstrong optionaltest evidence with provenance
A declared variant that adds checks and never removes baseline ones, in a referee-owned filereference-shopscope declaration, test-plan profile
Local completion and deployment evidence reported as two outcomesstrong optionaltest evidence versus operational acceptance

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Note: whether an agency asks suppliers for an SBOM is the agency’s choice (see the status box above); the status shown is for building and comparing one as engineering practice.

Key takeaway

  • Acceptance is a stranger running one command and reproducing every non-live check, not the author’s own green run.
  • The variant you declare adds checks and never removes one, and the file that declares it is out of the agent’s reach.
  • “Local exercise: COMPLETE” and “Deployment evidence: COMPLETE” are two outcomes, never combined; deployment counts only a pass file with a named person, and a NOT RUN never counts.
  • What cannot be tested without a live service or a restore is named, run by a person, and attached; it is never mocked quietly.
  • Every shipped behavior has a rule in modernization/rules.md, a test that exists, an evidence item whose hash recomputes, and a named human.
  • The reviewer’s job becomes following a trace, not trusting a summary.
LEADERSHIP DECISION
Name the owner and the date for each environment test, decide whether the cold run is a merge requirement, and decide that "Local exercise: COMPLETE" is never reported as readiness to deploy.
PRACTITIONER ACTION
Declare variant.json, add verify:cold, the trace file and its checker, then hand the repository and README to someone who has not seen it.
SUCCESS MEASURE
Assessor-hours: a reviewer follows any behavior to its rule, test, evidence and owner in minutes, with no meeting to explain it.

Legend: LEADERSHIP DECISION = what a leader decides · PRACTITIONER ACTION = what the engineer does next · SUCCESS MEASURE = how you know, in calendar time, assessor-hours or cost, contract or revenue risk, or audit finding avoided

Search lessons