intellimetrics Learning
Grounded Answers From Documents Module 5 · Authorize the Answer

Who Can Ask What

Last reviewed

Advanced

What you'll learn

~18 min
  • Explain why answer-level sensitivity can exceed every cited passage's
  • Set a serving mode per question class, sized to the cost of a wrong or leaked answer
  • Route the aggregation ruling to authority instead of deciding it in configuration
ℹLeadership brief

What it is: two decisions per question class, made by named owners before serving: who may ask, and whether answers go direct or as drafts for a human checker.

What it buys: the residual risk that Module 4 said survives every check lands somewhere chosen — on a checker for consequential classes, on a bounded audience for sensitive ones — instead of wherever the next question takes it.

What to fund: checker time for the classes that need it. That cost is the honest price of serving answers whose consequences exceed the evals’ guarantees, and it is much smaller than it sounds once the direct classes are carved off.

Prompt first: draft the serving matrix

For each question class [paste classes, tiers, and eval results]:


1. WHO MAY ASK - from 2.3's class tiers, now enforced at the serving
   surface, not just inside retrieval.
2. SERVING MODE - recommend one of:
     DIRECT    the asker gets the answer, citations attached
     DRAFT     the answer goes to a named checker who performs the
               4.1 hand check before it reaches the asker
     BLOCKED   the class does not serve until [condition]
   Base the recommendation on: cost of a wrong answer acted on,
   cost of a leaked answer, and the class's eval history - not on
   what feels impressive to ship.
3. THE AGGREGATION FLAG - which classes could combine individually
   permitted passages into an answer more sensitive than any of
   them? Flag; do not resolve.


Present as a table for the service owner to amend and sign - this
is their decision, prepared by you, not made by you.

The aggregation problem, transposed

If you took Building Trustworthy Data Products, you met the rule that two permitted inputs can join into something neither was cleared for. Answers do this too, and more fluently than any join. Ten field notes, individually all-staff, each mentioning one substation’s intermittent faults; the question “which substations have recurring unresolved faults?” produces an answer that is, functionally, a vulnerability map of the distribution system — a document nobody wrote, at a sensitivity no passage carries, assembled on demand.

Passage-level tiers cannot catch this: every citation checks out at all-staff. The control has to sit at the class level — which is why 1.3 bounded classes and why the prompt’s step 3 flags the classes whose answers aggregate. The ruling itself — is the aggregate a new sensitivity, and whose call is that — goes to the same authority that owns classification questions, with the flag and an example answer attached. The builder’s job is to notice and route, not to decide in a config file.

The serving mode is where residual risk lands

Module 4 ended with a promise: bound the consequence, don’t deny the residual. This is the bounding. Three modes, chosen per class by the cost of failure, not the maturity of the technology:

Direct — for classes where a wrong answer costs a recoverable nuisance and the evals are clean. The failure-history class: a wrong answer costs a supervisor a cross-check against a work order, the same check they perform on human answers.

Draft-for-checker — for classes where a wrong answer reaches a regulator, a customer, or a contract. The obligations class serves drafts: the answer plus citations lands with the contracts manager, who performs the ninety-second hand check before anything leaves. The system still saves the hours of finding; the human still owns the assertion. This mode is not a consolation prize — for consequential classes it is the designed end state, and teams that treat it as a stepping stone to direct are re-litigating 4.4’s limits with wishes.

Blocked — for classes that failed a gate, or whose aggregation ruling is pending. Stated as blocked, with the unblocking condition named, so it is a status and not a quiet death.

⚠The mode is per class, and the pressure will be to flatten it

Direct mode’s convenience creates constant pressure: “the failure-history answers are instant, why does contracts take a day?” Because a wrong failure-history answer wastes a truck roll and a wrong obligations answer waives a claim — the modes differ because the consequences differ, and the matrix records that reasoning so it survives the third time the question is asked. Flattening to all-direct is undoing 4.4 by convenience; flattening to all-draft buries the checkers and kills the capability by queue.

Stop and escalate when a class’s aggregation flag is raised and no one holds authority over the aggregate’s sensitivity — the passage tiers had owners, but the synthesized answer is new territory, and inventing its classification in a serving config is the same unauthorized act as 2.3 warned against, one level up.

KNOWLEDGE CHECK

Every passage cited in an answer is all-staff, every citation opens, every claim grades supported. The answer itself maps which substations have recurring unresolved faults. What has the passage-level control missed?

Key takeaway

Answers aggregate: individually permitted passages synthesize into artifacts no passage’s tier anticipated, so who-may-ask is enforced per class at the serving surface, and aggregation flags route to classification authority instead of dying in a config file. The serving mode is where Module 4’s residual risk lands by design — direct where failure is a nuisance, draft-for-checker where it reaches a regulator or a contract, blocked with a named condition otherwise — recorded with its reasoning, because the pressure to flatten the matrix arrives weekly. Lesson 5.2 turns to the record the serving surface produces, which is a corpus of its own.

LEADERSHIP DECISION   sign the serving matrix - which classes answer
                      directly, which draft for a checker - and fund
                      the checker minutes the consequential classes
                      cost
PRACTITIONER ACTION   prepare the matrix with eval history attached;
                      flag aggregation classes to classification
                      authority; never resolve sensitivity in config
SUCCESS MEASURE       every class serves in a signed mode; zero
                      aggregation flags resolved by the builder; the
                      matrix survives its weekly flattening pressure
Search lessons