Compare Candidates on One Contract

Before the detail — Decision: compare two candidates only on one contract. Outcome: a version standard the record can defend. Artifact: the comparison table with its gaps recorded. Status of what follows: binding where the contract carries M-25-22’s term; common baseline elsewhere.

Prompt first: draft the comparison shell

Here are two results files from the harness, one per candidate:
results/candidate-A.eval.json and results/candidate-B.eval.json,
both produced by the documented command against the same config.

Draft a COMPARISON TABLE SHELL, filling in nothing you cannot read
from the files:
  - one row per gate from the config, in config order
  - columns: candidate A pass/fail, candidate B pass/fail, n (the
    cases the gate ran), protocol (config hash and the environment
    lines), environment (mock / local model / target system), run id
    and timestamp
  - a CONTRACT CHECK line above the table: same set, same n, same
    protocol, same environment - state which held, from the files,
    and which you could not confirm
  - a GAPS section: what this comparison could not observe - left
    NEEDS-OWNER
Do not say which candidate is better. Pass/fail comes from the
files; every judgment is NEEDS-OWNER.

The agent reads the harness’s JSON precisely and lays out a table faster than anyone; what it cannot know is what the gaps mean for Meridian, and the prompt stops it from guessing. The shell is the agent’s; the decision, and the honesty about what the decision rests on, is the owner’s.

One contract, two candidates

Two kinds of comparison arrive at the release owner’s desk. The first is a provider’s current model version against its proposed replacement — the version boundary from 2.2, crossed on purpose. The second is two configurations of your own system: 3.1’s classifier at two thresholds, where the same seeded scores, thresholded in two separate runs, produced a FAIL at the default and a PASS at the lower cut. Either way the contract is the same.

Same set: both candidates run the sealed set, and if the set was exposed to tune candidate B, it is no longer sealed for B. Same n: fewer cases for one candidate means its pass line means something different — a gate passed on half the set is not the same gate. Same protocol: the config hash, the flags, and no cache, because a cached run compares a memory to a model. Same environment: the mock, the local model, and the target are evidence levels, and a candidate run on one level against a candidate run on another is two experiments, not a comparison. Break any of the four and you have two true statements about two different things.

Record the decision with its gaps

The candidate decision is recorded with three things: which candidate, on which evidence, and what the comparison could not observe. The gaps are not an apology; they are the part of the record a reviewer reads first — Data Products 1.4 (a separate training in this series): an acceptance test checked against something independent of what you built — because the person who did not run the comparison is the one who will notice what it skipped.

Typical gaps: one attempt per case, when Lesson 4.2 will show that retries change the picture; slices the set does not cover; questions outside the set; behaviour on the target under load when the run was on a quiet endpoint; cost. Each is written down as a gap with a NEEDS-OWNER next to it, and a gap the owner decides to accept is accepted in writing. The comparison screens the candidates — Grounded Answers 4.4: a screen, not a verdict — and does not certify the winner. A gap written down is a known limitation; a gap discovered later is a finding.

The version standard, and where rollback lives

The rule this lesson takes from policy is short. The commercial starting practice is a written acceptance bar for a version bump — a team’s own rule, waivable under deadline. The federal delta: M-25-22 (OMB — the White House budget office that binds agencies — April 2025) directs agencies to require a performance standard the new version must meet before it is deployed, with rollback if it fails — binding on the vendor once the contract carries the term. The record carries the standard, and the standard is the gate matrix’s pass lines, named: candidate B is approved to replace A if it passes every gate A passed, on the same set, under the same protocol. Where Meridian’s task order incorporates that term, the standard is what the contracting officer holds the vendor to; where it does not, it is still the only version of “better” the record can defend.

What this lesson does not teach is the rollback itself, because it is already taught: DevSecOps 4.3, where the rollback decision is made in advance by a five-minute rule; Cloud Modernization 4.3: promotion with preconditions. The release owner’s record names the standard; the deployment pipeline enforces the rollback, and the two are joined by the record’s evidence link, not by this lesson. Nor is there a drift lecture here — Grounded Answers 6.2 owns the re-evaluation triggers, and Lesson 6.1 adds the provider’s side. The handoff artifact is the recorded standard plus the comparison table on the record; what is not equivalent is enforcement — the commercial bar is the team’s to waive, the federal one is a task-order term the contracting officer holds the vendor to, with rollback required rather than offered. A version standard written before the new version arrives is a clause; one written after it arrives is a negotiation.

Stop and escalate when the two candidates cannot be run on the same environment — the current provider version was retired before the comparison ran, or the proposed one exists only in the vendor’s sandbox. Record that the comparison is one-sided, route the missing side through Lesson 3.4, and do not let a one-sided comparison stand in for a decision.

Practice status — among teams that replace a model or service version under a release owner, commercial and federal

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Key takeaway

Two candidates are comparable only on one contract: same set, same n, same protocol, same environment — break one and you have two true statements about two different things. The candidate decision is recorded with its gaps, because the reviewer who did not run it reads the gaps first, and the comparison screens rather than certifies. The version performance standard lives on the record as the gate matrix’s named pass lines; the rollback lives in the deployment pipeline where it was already taught, and the triggers live where they were. Lesson 3.4 turns to the systems you cannot run anything against.

LEADERSHIP DECISION   approve a replacement version only against a
                      standard written before it arrived - the gate
                      matrix's pass lines, on the sealed set
PRACTITIONER ACTION   hold both candidates to the same set, n,
                      protocol, and environment; record the decision
                      with its gaps and a NEEDS-OWNER on each
SUCCESS MEASURE       zero version changes deployed without a recorded
                      comparison and a named standard - an audit
                      finding avoided