intellimetrics Learning
Secure Software Delivery for Federal Environments Module 3 · Evidence and Assessment

Collectors and Verdicts

Last reviewed

Intermediate

What you'll learn

~18 min
  • Design catalog-driven plugin collectors with a two-stage assessment pipeline
  • Emit verdicts in 800-53A vocabulary with strict boolean-AND rollup
  • Account for coverage honestly - including what your automation cannot grade

From contract to collector

Lesson 2.1’s catalog declared an evidence contract per control — automated feeds, operator artifacts, queries. This module builds the machinery that honors it. The architecture, from the reference system this training draws on (a genuinely operating 2026 collector, not a whiteboard): a plugin registry of collectors, driven by the catalog. Each collector knows how to fetch one kind of evidence (an identity export, a config snapshot, a query result); which collectors run, against what scope, is read from the control YAML — the catalog stays the single source of truth, and adding evidence for a control is a data change, not a code change.

Assessment then runs in two stages, and the separation is the design:

STAGE 1: PREFLIGHT PROBES     can we even look? - endpoints reachable,
                              permissions valid, evidence paths present.
                              Failures here are reported as COLLECTION
                              problems, never as control failures.


STAGE 2: ROLLUP EVALUATOR     given evidence that was actually collected,
                              evaluate each check and roll up per control.

Why two stages instead of one: a collector that can’t reach its endpoint has learned nothing about the control. Collapsing “we couldn’t look” into “it failed” poisons the results in one direction; collapsing it into “it passed” (the silent default of many home-grown scripts) poisons them fatally in the other. Stage separation makes “couldn’t look” a first-class outcome with its own reporting lane.

Speak the assessor’s language, roll up strictly

Verdicts are emitted in 800-53A-aligned vocabulary — satisfied, other-than-satisfied, plus an internal partial state that maps to other-than-satisfied (with detail) in strict 800-53A terms, where the finding vocabulary is binary — because Module 4’s consumers (assessors, and the systems of record from Lesson 2.3) think in those words, and a bespoke pass/fail vocabulary just creates a translation layer where trust leaks out.

The rollup rule is strict boolean AND: 24 of 25 checks passing yields partial, not satisfied. No percentage thresholds, no weighted scores. This feels harsh until you see what it buys: a satisfied from this system means something — every declared check, across the declared scope (2.1’s mandatory boundary), passed. The moment 96% rounds up to satisfied, every satisfied in the package inherits an asterisk, and the assessor goes back to checking everything by hand — which was the thing the machinery existed to prevent. Related vocabulary note: the Zero Trust training’s honest-grading lesson teaches a five-value alignment scale for control-to-intent grading (in short: each value names who owns the gap); this lesson’s vocabulary grades evidence-to-control. Different axes — strong programs run both.

Completing the taxonomy, failures are loud and typed: evidence-path failures (a missing artifact, an unreachable store) land in a warning band that never silently fails a run; tool faults fail the run itself — a crashed collector is a broken instrument, and a broken instrument reporting verdicts is worse than no report.

Honest coverage accounting — the part most shops skip

Now the numbers that make this lesson real. The reference system, at its 2026 maturity: of an authorization register of roughly 209 control-rows, about 165 are technical rows covered by automated collection … but of the ~84 catalog controls behind them, only 35 are fully graded by the two-stage evaluator — the rest run in an inventory-review mode (evidence collected, human evaluates). The remaining ~44 rows are manual by nature — policy, personnel, narrative — and within that manual bucket, some 29 are answered by inherited citations rather than local evidence. And two warts the operators say out loud: operator-uploaded evidence is stored but not yet a verdict input, and an assessment-report builder exists but was never shipped — an orphaned component the program routes around.

Those numbers aren’t a confession — they’re the deliverable. This is a system that can answer “what does your automation actually grade?” with a table instead of a vibe. That’s rarer than the automation itself, and it’s the design requirement this lesson adds: your coverage accounting is a first-class artifact, versioned beside the catalog, current release always citable. The warts are also your design checklist — every “stored but not evaluated” input and orphaned component in your own machinery belongs in that table, because the alternative is your assessor discovering them for you.

Prompt first: build the accounting before the collectors

From our control catalog (all YAML in controls/), generate the coverage
accounting table BEFORE we write any new collectors:
1) per control: evidence contract entries vs collectors that actually
   exist in collectors/ - classify as GRADED (evaluator consumes it),
   COLLECTED-ONLY (gathered, no verdict logic), or ABSENT;
2) flag every input that is stored but feeds no verdict - the
   collected-only trap - with the file that stores it;
3) emit the table as coverage.md with counts in the header, plus a
   ranked build-next list: which ABSENT collectors would move the most
   controls from manual to graded, cheapest first.

Running the accounting first inverts the usual failure: most programs build collectors opportunistically and discover their coverage story at assessment time. The table makes coverage a managed number that only moves up — and the build-next ranking turns it into a roadmap.

💬Automation theater is the failure mode

The worst version of this machinery is the one that looks complete: collectors everywhere, dashboards green, and - underneath - percentage rollups rounding partial to satisfied, couldnt-look collapsing into pass, operator evidence uploaded into a folder no evaluator reads, and an assessment builder that never shipped still on the architecture diagram. Every element of this lesson - stage separation, strict AND, typed failures, the coverage table, warts said out loud - exists to make the machinery UNABLE to flatter you. An evidence system’s value is exactly the trust an external skeptic can place in its outputs; optimize for that skeptic, not for the dashboard.

KNOWLEDGE CHECK

A collector run reports: AC-2 - 24 of 25 checks pass; the 25th check could not reach its endpoint (auth token expired). What should the system report for AC-2, and why?

Key takeaway

Catalog-driven plugin collectors; preflight separated from evaluation so couldnt-look never masquerades as pass or fail; verdicts aligned to the assessor’s 800-53A vocabulary with strict-AND rollup (24/25 is partial, mapping to other-than-satisfied); failures typed and loud. And the requirement most shops skip: coverage accounting as a versioned, citable artifact — graded vs collected-only vs absent, warts included. Build the table before the collectors, and let it be your roadmap. Next: the store all this evidence lands in, and the trust model that has to survive a hostile reading.

Search lessons