Impact, Baseline, and Benefit

Before the detail — Artifact: section 1 of the record. Status of what follows: binding for federal high-impact use (M-25-21); ISO/IEC 42005 guidance elsewhere.

Prompt first: draft section 1 from the register entry

Here is the register entry for MU-AI-002, the Feeder Outage Predictor
[paste the ai-inventory.yaml entry].

Draft SECTION 1 of its release-decision record under these headings,
in this order:
  1. Purpose - the decision or task, for whom
  2. Current practice it replaces - what dispatchers do TODAY
  3. Expected benefit and its measure - unit, sampling rule, reader
  4. Data fitness - what the model was trained on versus what it
     will see in production; name every gap you can infer
  5. Impact of use AND impact of non-use, in the same units
  6. Effects on the people at the other end - privacy, civil rights
     and liberties, safety
  7. Direct cost; cost of a false alarm; cost of a missed feeder
  8. Reassessment schedule - triggers plus a fixed date

Do NOT invent a baseline, a benefit figure, a cost, or a training
window. Mark each NEEDS-OWNER with the question L. Tran must answer.

The draft is useful because it asks the eight questions in order and leaves the answers blank. Every figure it could have filled in is a fact only Distribution Operations holds, and a section 1 with invented figures is a slogan with a table around it.

A benefit without a baseline is a slogan

“Ranks feeders by outage likelihood” is the purpose, and purpose is the easy line. The line drafts skip is what dispatchers do today: at Meridian, a storm desk pre-positions crews from last season’s outage map and a supervisor’s judgment. That practice is the baseline, and the benefit exists only as a comparison against it — crew hours pre-positioned to feeders that did fail, versus the storm desk’s record over the same storms.

State the measure before the benefit. “Faster restoration” is not a measure; “customer outage minutes on pre-positioned feeders, sampled per storm, read by the operations manager” is — a unit, a sampling rule, and a reader. Module 6 holds the system to that measure after release. A record that cannot say what “better” means in a unit the owner already tracks cannot tell a leader, a year on, whether the money moved anything.

Data fitness: what the model saw versus what it will see

This element is new here, and no evaluation gate supplies it. The predictor was trained on the outage history Meridian kept: the feeders that existed then, asset-age fields as recorded then, the weather of that window. It will see this season’s weather, feeders rebuilt since, and asset-age fields a replacement program has reset. Data fitness is the named gap between the two pictures, written before the first ranked list reaches a dispatcher — plus which of those gaps Module 3’s sealed set actually exercised. Where the set did not, say so; that bounds the claim rather than hiding it. For a vendor-hosted system whose training data is undisclosed, record the omission from the evidence packet and what the owner will watch instead. An unstated training window is an audit finding waiting: an evaluation that cannot be shown to cover what the system now sees.

Use, non-use, and the people at the other end

Every draft lists the impact of deploying. Almost none lists the impact of not deploying, and the omission tilts every gate toward rejection by default. The predictor undeployed has a cost — crews at the yard, the storm desk guessing, outage minutes that pre-positioning would have cut still paid by customers. Write both in the same units, so a “no” is a choice with a stated price.

Then the row where harms actually live. Module 4’s attack taxonomies map ways to break a system; they say nothing about what a working system does to a person. This row does: privacy (whose data is in the features), civil rights and liberties (a feeder ranked low gets crews later — if low-ranked feeders track a neighborhood, that is a fairness question with a name), and safety (a crew sent to the wrong feeder in a storm). Where personal data is involved, the PIA (the privacy impact assessment) is the companion, not the substitute. Close with cost — the batch job and the reader’s hours — and the two halves of failure cost kept apart, a false alarm and a missed feeder, because different people pay each. Then the reassessment schedule: quarterly retraining is an evidence-invalidating trigger in its own right, beside a fixed date after which section 1 is stale regardless.

The seam: what the federal list adds, and what it does not

Meridian operates one AI use case on behalf of a covered agency under a contract that incorporates the federal requirements, so the federal list binds that system and is the reference for the other nine. M-25-21 (OMB’s April 2025 memo governing federal AI use — OMB being the White House budget office that binds agencies) names the impact-assessment elements for a high-impact use: purpose and expected benefit with metrics; a data quality summary that, where applicable, describes information in the data about classes protected by Federal nondiscrimination laws; potential impacts on the privacy, civil rights and civil liberties of the public, of using AI and of not using it; a reassessment schedule; a cost analysis; independent review by “an independent reviewer within the agency who has not been involved in the development”; and risk acceptance “supported by a signature from the individual accepting the risk.” This lesson’s eight headings were built from that list.

The commercial starting practice is a corporate impact-assessment template. The one most teams have seen is a large vendor’s — proprietary, offered “for reference only,” readable but not adoptable.

ISO/IEC 42005:2025 (the per-system AI impact-assessment guidance) is the crosswalk between such a template and the federal list, and ISO/IEC 42001 Annex A control A.5 is the control the finished section supports evidence for. Two open templates can be adapted outright: Canada’s Algorithmic Impact Assessment tool is MIT-licensed; the UK’s Algorithmic Transparency Recording Standard template is under the Open Government Licence.

The handoff artifact is the completed section 1 — one text serves the commercial register and the federal package. Not equivalent: a commercial impact assessment is voluntary and unpublished; the federal one was due within 365 days of the memo, by April 3, 2026, and a high-impact use that does not meet it “must safely discontinue.” Enterprise policy and risk appetite belong to the leadership training planned for AI adoption; this section decides one system.

Stop and escalate when nobody can state the current practice the system replaces. Without a baseline the benefit has no measure and the gate nothing to weigh; the fix is a scoping decision by the system owner, not a figure the agent fills in so the section looks complete.

Practice status — among organizations that gate AI releases, commercial and federal

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Section 1 covers the memo’s elements A through E: purpose and expected benefit with its metric; the data — including the model’s capability, how the data was collected and prepared, the fields that touch protected classes, and the summary across the system’s lifecycle; potential impacts; the cost analysis; and the reassessment schedule with the procedure that follows a significant modification. Elements F and G — the independent reviewer’s findings and the signed acceptance — are completed in section 7, not here.

Key takeaway

Section 1 makes the decision weighable: a purpose, the practice it replaces, a benefit with a unit and a reader, the gap between what the model saw and what it will see, the cost of deploying and of not deploying in the same units, the effects on the people at the other end, the two halves of failure cost, and a date for asking again. The federal list names and requires these elements; the commercial template is voluntary and unpublished; one section serves both. The agent drafts the headings, the owner supplies every figure. Lesson 5.2 assembles the record around it.

LEADERSHIP DECISION   fund the owner's hours to state the baseline and
                      the benefit measure; refuse a section 1 that
                      names a benefit with no unit
PRACTITIONER ACTION   draft the eight headings with the agent, fill
                      every figure from the owner, write non-use in
                      the same units as use
SUCCESS MEASURE       zero release decisions weighed against a section
                      1 with an unstated baseline - an audit finding
                      avoided on every federal-facing system