Evaluation by System Type

Before the detail — Artifact: the gate matrix per system type with named label owners. Status of what follows: M-25-22 guidance on sealed data; common baseline elsewhere.

Prompt first: draft the gate matrix from the register

Here is the AI system register [paste data/ai-inventory.yaml] and the
frozen evaluation set for each system that has one [paste].

Draft a GATE MATRIX, one block per system:
  - system type: classifier / generator / code assistant / agent /
    provider-controlled (no model access - Lesson 3.4's route)
  - the gates that type needs, each a SEPARATE pass statement, never
    one blended score: classifier = recall at threshold, calibration,
    every subgroup slice; generator = planted fact / planted negative
    / directive probe; code assistant = gates on the pipeline
  - the costly error, in plain words ("a missed outage")
  - label owner: whose records say what the right answer was
  - sealed set: the portion withheld, who holds it, and a line
    stating the provider never sees it
  - a THRESHOLD column, every row NEEDS-OWNER

Do not invent a threshold, costly error, or label owner. Mark each
NEEDS-OWNER.

The agent knows the shape — what a confusion matrix is, which gates a generator needs. It does not know that a missed outage means a crew dispatched too late, or that L. Tran’s outage records are the only labels that count. Those come from the owner; the matrix leaves blanks where they go.

The set arrives; the orchestration is new

The frozen set is a given — Grounded Answers 1.4 (a separate training in this series) freezes the question set before the build. Every system that needs one has one, written by its label owner and sealed in part. What that training could not teach, having one system, is that gates differ by kind of system: a matrix that treats an outage classifier like a chatbot passes the wrong thing.

The ancestor is DevSecOps 3.5: the scan matrix, per environment. Here rows are register systems, columns are gates by type, and every row has an owner and a pass statement or it does not ship.

The classifier: a threshold is a decision

MU-AI-002, the feeder outage predictor L. Tran owns, returns a score per feeder. Nothing happens until someone picks the score above which a feeder is flagged — the threshold — and that choice is the release owner’s, not the model’s. Every other number moves with it. The classifier script in the substrate scores a seeded synthetic set, so every learner sees the same numbers; at the default threshold, on this run:

MU-AI-002 Feeder Outage Predictor - evaluation at threshold 0.5 (seed 4471, n=600)
  confusion  tp=108 fp=46 fn=143 tn=303  precision=0.701  recall=0.43
  slice rural      n=257  precision=0.843  recall=0.547
  slice urban      n=343  precision=0.508  recall=0.289
  slice age 0-10   n=203  precision=1.0    recall=0.1
  slice age 11-25  n=194  precision=0.697  recall=0.315
  slice age 26+    n=203  precision=0.687  recall=0.669
  calibration ECE=0.0454   rural-urban recall gap=0.258
  gate  FAIL recall >= 0.70 (missed outages are the costly error)
  gate  PASS expected calibration error <= 0.05 (dispatchers read the score as a probability)
  gate  FAIL rural-urban recall gap <= 0.10 (the model must not quietly under-serve either population)
  verdict FAIL - gate failure observed on this set
legend  tp flagged outage · fp false alarm · fn missed outage · tn correct clear

Say first what those numbers are: a seeded synthetic set scored by a stand-in — bench-level evidence that teaches the method and supports no decision about MU-AI-002. Only the target system’s run on the sealed set is release evidence (Lesson 3.2 states the three levels as a rule).

The confusion matrix (the four counts the legend names) is read through two ratios. Precision (the share of flagged feeders that really failed) is respectable; recall (the share of real outages the model caught) is not, and recall is the gate because the costly error is a missed outage — a crew not pre-positioned, a longer restoration. Calibration passes: ECE (how far predicted probabilities sit from observed rates) is inside the example bound, which matters because dispatchers read the score as a probability. The slices fail on their own — urban recall sits far below rural, and the gap gate exists so the model cannot quietly under-serve one population behind a tolerable overall figure. Rural/urban and asset-age band are the slices a dispatcher would ask about.

Lower the threshold — same scores, same seeded set, a separate thresholded run:

MU-AI-002 Feeder Outage Predictor - evaluation at threshold 0.3 (seed 4471, n=600)
  confusion  tp=201 fp=168 fn=50 tn=181  precision=0.545  recall=0.801
  calibration ECE=0.0454   rural-urban recall gap=0.069
  verdict PASS - no gate failure observed on this set at this threshold
legend  slice and gate lines omitted here; the script prints all three gates PASS

Nothing about the model changed. Recall rose and precision fell: more false alarms, which cost crew-hours, against fewer missed outages, which cost restoration time. Which cost Meridian carries is L. Tran’s decision, written into the matrix as the threshold. The script’s gates are the release owner’s examples, not law; its last line is the only claim the run supports. A threshold nobody chose is a reliability commitment nobody signed.

Generator, code assistant, agent

MU-AI-004, P. Delgado’s answering system, is graded by planted facts, planted negatives, and directive probes as separate gates — Grounded Answers 4.2: refusals are cases too. A planted fact fails when the cited passage lacks the claim; a planted negative fails when the system answers what the corpus cannot; a directive probe fails when quoted text is obeyed. Blended, a system that fails every refusal reads as “mostly fine”; separated, the failure has a name and an owner.

MU-AI-003, the developer code assistant, is a subscription; nobody at Meridian can evaluate its model. Its gates sit on the pipeline: secret leakage in completions (does a completion reproduce a credential pattern from the repository), license-tainted suggestions (does a suggestion reproduce code under terms the project cannot carry), and the repository allowlist — the register excludes the SCADA integration repositories, and the gate asks whether that is enforced or merely written down. The label owner is Enterprise IT; the “set” is a list of probes run in the build.

An agent needs trajectory evaluation (grading the sequence of actions, not the final answer). It is named so the matrix has a row; the row says “addendum,” and Lesson 4.3 and the next training carry it.

Labels, sealed data, and the honest scope

A label is the record of what actually happened, so its owner keeps that record: L. Tran’s outage history for MU-AI-002, P. Delgado’s corpus and hand checks for MU-AI-004, Enterprise IT’s pipeline for MU-AI-003, R. Okafor’s customer-experience team for MU-AI-001, whose route is Lesson 3.4. A row with no label owner is a row with no evaluation, whatever the vendor’s table says.

The commercial starting practice for sealed data is a held-out set the vendor never sees, kept by the label owner. The federal delta: M-25-22 (OMB — the White House budget office that binds agencies — April 2025) directs agencies to require ongoing testing with agency-defined validation data that “should not be accessible to the vendor,” and vendor tests “detailed enough to be independently verified or reproduced, if practicable.” The handoff artifact is a custody record for the sealed set — who holds it, when it was last exposed, which runs used it. Not equivalent: the commercial held-out set is hygiene a team can waive under deadline; the federal one is a contract term with a cadence, and since the memo gives “quarterly or biannual” only as examples, Meridian’s cadence is a decision to record. Where the task order incorporates those terms, the custody record is what the contracting officer asks to see.

NIST AI 800-2 (initial public draft, January 2026) is the current federal evaluation guidance — voluntary practices, in its own words — scoped to automated benchmark evaluations of language models and agents. It supports evidence for the generator row and says nothing about the classifier’s threshold or the pipeline gates; citing it for the whole matrix claims what the document does not. Every row’s pass statement has one shape — Grounded Answers 4.4: a screen, not a verdict — and the custody record is the difference between “we tested” and “we can show what the vendor never saw.”

Stop and escalate when a system has no label owner, or when the only labels available were produced by the provider — labels the vendor wrote cannot seal anything. That is a register problem for the system owner, or for a provider-controlled system Lesson 3.4’s route, not a gap to fill with the vendor’s benchmark table.

Practice status — among organizations running more than one kind of AI system, commercial and federal

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Key takeaway

The frozen set arrives; this lesson adds orchestration by type. A classifier is evaluated at a threshold the owner chooses, its confusion matrix, calibration, and slices read as separate gates with the costly error named; a generator gets three pass lines; a code assistant is gated on its pipeline; an agent’s row says “addendum.” Every row names a label owner and a sealed portion the provider never sees, and the only pass statement is “no gate failure observed on this set at this threshold.” Lesson 3.2 turns to the workbench that runs these gates — and to what each level of its evidence may prove.

LEADERSHIP DECISION   fund a label owner per system and a custodian for
                      the sealed set; set every threshold yourself, as
                      a recorded decision
PRACTITIONER ACTION   draft the gate matrix by system type from the
                      register; separate pass lines per gate; leave
                      thresholds and costly errors NEEDS-OWNER
SUCCESS MEASURE       zero systems on the register without a named
                      label owner, a sealed portion, and separate pass
                      lines - an audit finding avoided