Grounded Answers From Documents Module 6 · Re-test Every Changed Dependency

Own It or Shut It Down

Last reviewed

Advanced

What you'll learn

~18 min
  • Walk the wrong-answer incident using only artifacts this training built
  • Write the stewardship record that keeps the service accountable after its builder leaves
  • Retire a QA service deliberately, because this capability degrades into confident wrongness
ℹLeadership brief

What it is: the accountability layer — who answers when a wrong answer is acted on, the stewardship record that survives staff turnover, and the retirement decision this capability needs more than most systems do.

What it buys: the wrong answer that eventually lands becomes a bounded incident with a named owner and a paper trail, instead of a credibility collapse. And the service gets to end on purpose, rather than degrading into a confident liar nobody remembers owning.

What to fund: an owner role with real authority over serving modes and shutdown, and the annual review that keeps the eval history honest.

The incident, walked

It happens the way 4.4 promised it would: the supervisor postponed a regulator replacement on the strength of an answer, the regulator failed, and the answer — it turns out — was wrong. The question now is not whether the system failed; it is whether the organization can answer four questions in an afternoon:

  1. What exactly was served? The answer record (6.1): question, answer, citations, serving mode, six version stamps. Not memory, not a re-run — the artifact.
  2. Was the system inside its license when it served it? The eval history: at that corpus version and configuration, had the class passed its gates? Was the serving mode per the matrix?
  3. Which half failed? 4.3’s routine, on the reconstructed answer: evidence missing (corpus), evidence misused (generation), or — the third row — both halves faithful to a corpus document that was itself wrong or stale.
  4. Who owns the fix? The routing table’s address column, which has been every escalation’s destination since Module 2.

Walked this way, the incident produces a diagnosis, a fix with an owner, and — critically — a defensible account: the service was inside a stated, bounded risk (4.4’s licensed sentence), the residual landed as designed (5.1’s serving mode), and the correction is specific. The alternative account — “the AI was wrong, we’re looking into it” — is what organizations say when these artifacts do not exist, and it is the beginning of the credibility collapse.

The uncomfortable branch: if question 2’s answer is no — the class was serving outside its evaluated license, a re-eval trigger never fired, a mode was quietly flattened — then the failure is the operation’s, not the technology’s, and the stewardship record below is what makes that distinction visible before an incident forces it.

The stewardship record

One block, kept with the service, in the pattern this site always uses — owner as a role, the holder tracked separately:

STEWARDSHIP - Meridian grounded QA service
OWNER (role) field engineering manager - serving matrix,
shutdown authority
CORPUS OWNER per manifest (2.4) - statuses, resolution rule
RECORD OWNER per 5.2 - the log's tier and retention
CHECKERS named, per draft-mode class
EVAL AUTHORITY who adjudicates the hand-check sample, per class
REVIEW annual: classes still earning their cost (5.3),
eval history unbroken, held-out sets replenished,
canary running
KNOWN LIMITS 4.4's licensed sentence, verbatim, per class -
plus the standing corpus-vs-world caveat
RETIRE IF (see below)

The KNOWN LIMITS line is this record’s distinctive entry: the licensed sentence travels with the service, so the claim made for it two owners from now is the claim the evals support, not the claim enthusiasm has grown.

Why retirement is sharper here

Every system this site teaches gets a retirement path. A QA service needs one more, because of how it fails when abandoned: a data product left unowned goes visibly stale — dashboards date-stamp their decay. An abandoned QA service keeps answering, fluently, at yesterday’s confidence, while its corpus rots, its refusals drift, and its eval license ages into fiction. Nothing about its output signals decay; that was 1.1’s warning, and abandonment weaponizes it.

So the retire-if conditions are concrete and standing:

  • the question classes stop earning their cost at review (5.3’s value question, answered honestly)
  • the corpus owner role goes unfilled past a stated grace — an ungoverned corpus is 2.1 undone
  • the eval cadence breaks and cannot be restored — serving without a live license is the one thing this training prohibits

And retirement itself follows the pattern the site always uses — announce, stop serving with a visible notice (never a silent death), archive the record and eval history intact (disputes outlive services), then remove. The archived answer record keeps answering question 1 of the incident walk for as long as old answers can surface.

Stop and escalate — the training’s last one — when the review finds the service healthy but the owner role vacant: a working system with no accountable human is not a stable state, it is the abandoned-liar failure on a delay timer, and filling the role or beginning retirement is a leadership decision that the review exists to force.

KNOWLEDGE CHECK

At annual review: the service answers fluently, users are satisfied, costs are stable. The eval history shows the last re-eval ran seven months ago; four corpus versions have shipped since, and the canary was disabled during a migration and never re-enabled. What is the service's actual state?

Key takeaway

The wrong answer will come, and the difference between an incident and a collapse is four questions answerable from artifacts: what was served, was it licensed, which half failed, who owns the fix. The stewardship record keeps the roles, the limits, and the licensed sentence attached to the service across staff turnover. And retirement is sharper here than anywhere on this site, because an abandoned QA service does not go visibly stale — it keeps answering, confidently, while its license ages into fiction. Own it with a live eval cadence and a named owner, or shut it down on purpose. That is the training.

LEADERSHIP DECISION fill the owner role with shutdown authority, or
begin retirement - a working service with no
accountable human is the failure on a delay
timer
PRACTITIONER ACTION keep the eval cadence and canary alive; walk any
incident by the four questions; retire with a
visible notice and an archived record, never a
silent death
SUCCESS MEASURE the four incident questions answerable in an
afternoon from artifacts alone; zero months of
serving outside a live eval license
Search lessons