AI Assurance: System Risk and Release Decisions Module 4 · Challenge the Evidence

The Attack-Evidence Plan

Updated

Advanced

What you'll learn

~18 min
  • Choose the attack classes that apply to a system from what it can do, not from a generic list
  • Turn a tester's finding into an adjudicated finding with a named reproducer and a rating
  • Write the release blockers from your own risk appetite before the first attempt is made
ℹLeadership brief

What it is: a plan, per system on the register, naming which attack classes apply, who reproduces and rates each finding, and which findings block release — decided by the risk acceptor before anyone has a result to argue about.

What it buys: attack evidence about your deployment rather than a vendor’s model, findings that are facts rather than claims, and a pass line written before the test so a result cannot move it.

What to fund: a few hours per system for the reviewer and system owner to make the applicability calls, plus reproduction time for every finding — the scarce, unglamorous input no tool supplies.

Before the detail — Artifact: the attack plan and the adjudicated findings sheet. Status of what follows: reusable guidance; M-26-04 enhanced tier only where the agency invokes it.

Prompt first: map the register to attack classes

Open your workspace agent on data/ai-inventory.yaml with the Lesson 1.3 consequence tiers filled in, switch into plan mode, and start with (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):

Map all ten systems in data/ai-inventory.yaml to candidate attack classes from OWASP LLM Top 10, OWASP Agentic Top 10, and MITRE ATLAS based on system capabilities. Mark unsettled applicability and blockers NEEDS-OWNER.

Check that the plan or draft covers:

  • System ID, consequence tier, and capability surface (answer only, read customer data, call tools, or take external actions)
  • Candidate attack classes mapped from the three public taxonomies, marked APPLICABLE, NOT APPLICABLE, or NEEDS-OWNER with the single architectural fact needed to settle each
  • Expected evidence artifacts (log lines, report rows, reproduction scripts) and the Meridian team responsible for reproducing findings
  • A blank BLOCKER? column reserved for the risk acceptor and release owner to complete before testing starts

If the agent decides severity ratings or release blockers on its own, follow up with: "Leave every applicability gap, severity rating, and release-blocker column marked NEEDS-OWNER."

The agent is good at the cross-product — ten systems against three taxonomies is tedious and it will not skip a row. It does not know whether the portal widget holds a session token; that fact belongs to R. Okafor’s team, and a plan built on a guessed fact tests the wrong system.

Which attacks matter depends on what the system can do

Three register rows. MU-AI-004 answers maintenance questions from documents with no internet egress; the worst it can do is say something wrong with a citation attached. MU-AI-001, the portal assistant, is internet-facing and touches customer PII; the worst it can do is say something about the wrong customer. A tool-using agent — Lesson 4.3’s subject — can act, and the worst it can do is whatever its tools can do.

The consequence tier from Lesson 1.3 sets how much evidence the gate demands; capability chooses the classes. An answerer gets its refusal and grounding attacked, a portal assistant its cross-customer leakage and system-prompt leakage, an agent its injection-through-content and excessive agency with the tools live.

Taxonomies are maps, not harms lists

Three public catalogues supply the vocabulary. OWASP (the community that publishes application-security top-ten lists) issued its 2026 LLM Top 10 in August 2026; it lists Excessive Agency third. OWASP’s Top 10 for Agentic Applications (December 2025) adds ten agent-specific categories.

MITRE ATLAS (a catalogue of observed attack techniques against AI systems) counts, as of its October 2026 release, 16 tactics and 129 techniques. Use them as coverage checks — did we consider each of these for this system? — and nothing more. They map what an attacker can do to the system, not what the system can do to a person. The resume screener, MU-AI-006, needs no attacker to harm the applicant it ranked fifty-first; that harm belongs in the impact assessment, Lesson 5.1. Keeping the two apart avoids the audit finding where a red-team report is offered as the impact assessment and rejected as one.

A finding is a claim until someone reproduces it

Scanners, testers, and vendors all produce findings, and each arrives as a claim: this prompt made the system do that. It becomes evidence when a named person at Meridian reproduces it, rates it on Meridian’s scale, and records the disposition — five fields per sheet row: finding, reproduction (who, date, steps), rating, disposition, link to the logged run.

Two earlier trainings own the rules the sheet leans on. DevSecOps 3.4 (a separate training in this series) gates on severity and gates harder in prod — the scale is yours, and it tightens as the tier rises. Zero Trust 5.2 grades the false positives — a detector that flags a refusal as a bypass has found nothing, and Lesson 4.2 shows one doing exactly that.

The blockers your risk appetite defines

Say it plainly: “no critical findings before release” is a policy someone at Meridian must write. Nobody published it as a standard. The risk acceptor decides which classes at which rating block a release for a given tier, and writes that down before the first attempt — a blocker written after the results is a blocker chosen to fit them.

The plan also states a stopping rule, because two of the eight lessons in Microsoft’s January 2025 paper Lessons From Red Teaming 100 Generative AI Products are “AI red teaming is not safety benchmarking” and “The work of securing AI systems will never be complete”: a benchmark measures a fixed thing on a fixed set, and a red team stops when you say so, not when it is finished. Name how many attempts per class, or which success ends the run, and claim only what was run.

The seam: commercial red team, federal disclosure

The commercial starting practice is a pre-release red team by people who did not build the system, mapped to the taxonomies, findings adjudicated as above. The federal delta comes from two directions. Buying: M-26-04 (the White House budget office’s memo on unbiased-AI purchase terms) lets an agency require, for a public-facing language model, an enhanced disclosure tier whose examples include red-teaming evidence, so a Meridian system serving its covered agency should expect the vendor’s evidence to be requested and have its own beside it. Testing: the UK AI Security Institute’s December 2025 trends report found universal jailbreaks for every system it tested, and the March 2026 analysis by the US Center for AI Standards and Innovation (CAISI, which now operates as the Center for Advancing Innovation and Standards for Super Intelligence, CAISSI) of a large-scale public red-teaming competition, run by Gray Swan with CAISI and the UK AI Security Institute (13 models, more than 400 participants, over 250,000 attempts), recorded at least one success against every model.

The handoff artifact is the attack plan plus the adjudicated-findings sheet. What is not equivalent: government testing of a frontier model tells you about the model — nothing about your safeguards, retrieval, tools, or customers’ data. Those findings are the reason to test your deployment, not a substitute.

Stop and escalate when the risk acceptor will not write the blockers before testing, or when nobody can be named to reproduce findings for a system on the full gate path. Either gap means results cannot become evidence, and running the attempts anyway buys a report nobody can sign against — a scoping decision for the release owner, not a gap to fill by testing harder.

KNOWLEDGE CHECK

A scanner report on MU-AI-001 shows one entry rated 'critical: hidden context extracted.' The vendor says it cannot reproduce it. The reviewer wants to block release today on the strength of the report. What does this lesson require?

Practice status — among organizations gating AI releases, commercial and federal

PracticeStatusAlso called
attack plan written from the system’s capability and tiercommon baselinethreat model for the AI use case
taxonomy coverage check (OWASP LLM, OWASP Agentic, ATLAS)common baselinethreat-catalogue mapping
adjudicated findings sheet (named reproducer, rating, disposition)strong optionaltriaged findings register
blockers written by the risk acceptor before testingstrong optionalpre-registered pass criteria
vendor red-team evidence requested at purchaserequired only where the agency invokes M-26-04’s enhanced tier; commercial: emergingthird-party test report as a purchase condition

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Key takeaway

An attack plan starts from what the system can do; the taxonomies check coverage and stay out of the impact assessment; a finding is a claim until a named person reproduces and rates it; and the blockers are a policy the risk acceptor writes before testing, because no standard wrote it for them. Government testing of a frontier model is evidence about the model — your deployment needs its own. Lesson 4.2 turns to reporting a result honestly: not one number, but a curve.

LEADERSHIP DECISION
write the release blockers per tier before any attempt is made, and name who reproduces findings - the plan is yours, not a vendor's
PRACTITIONER ACTION
map each register row to attack classes from its capability; adjudicate every finding with reproducer, date, rating, disposition
SUCCESS MEASURE
zero release decisions carrying an unreproduced finding as a blocker or a vendor's model test as deployment evidence - the audit finding avoided
Search lessons