intellimetrics Learning
Grounded Answers From Documents Module 6 · Re-test Every Changed Dependency

The Answer Record

Last reviewed

Intermediate

What you'll learn

~18 min
  • Stamp every answer with the six versions that produced it
  • Reconstruct a disputed answer months later from its record alone
  • Pass the stranger test: the investigation works without the person who built the system

Prompt first: stamp the answers

Extend the serving record from 5.2: every answer carries the six
versions that produced it -


  corpus_version      from the manifest (2.4)
  extraction_run      which extraction produced the indexed text
  index_build         which index served the candidates
  policy_version      resolution rule + serving matrix in force
  prompt_version      the answering contract's version
  model_id            the answering model, exactly as invoked


Then write the reconstruction query: given an answer id, return the
question, what retrieval offered, what was cited, all six versions,
serving mode, and - for drafts - who released it.


Test it on a real answer from last week, and tell me which fields
you could NOT populate. Missing fields are the finding: each one is
a question the eventual dispute will ask and the record cannot
answer.

The question that will arrive

Some months from now, the contracts manager will forward an answer the service gave in March and ask: why did it say this? Between March and the question, the corpus took two version bumps, the prompt contract was tightened once, and the model was updated by its provider. Ask the live system the same question today and you will get a different answer — which proves nothing about March.

The previous training met this exact moment for numbers and answered it with the run record: logic, inputs, and parameters, pinned per run. The transposition to answers is direct, but the load is heavier here in two ways. A pipeline had one version that mattered per run; an answer has six independently moving dependencies, and any one of them changes the output. And a wrong number was wrong quietly, while a wrong answer was fluent — the dispute will arrive with the answer’s own confident prose as evidence against you.

The six stamps make the March question mechanical: reconstruct what the model saw (retrieval candidates at that corpus version), what governed it (policy and prompt versions), and what wrote it (model id). Then 4.3’s routing runs on a months-old answer exactly as it runs on this morning’s: was the evidence there, was it used faithfully — which half, whose fix.

The stranger test

The standard, inherited whole from the site’s evidence discipline: someone who has never met you must be able to follow the record to a verdict. For the answer record, the stranger needs:

  • the answer id → the reconstruction query’s full output
  • the corpus manifest at that version → what the system could have known
  • the eval results on record for that version → what was verified at the time
  • the serving matrix then in force → whether this asker, this class, this mode were legitimate

If any step requires asking the builder what things were like in March, the record has failed the test — and the builder-shaped hole is exactly what Module 6 exists to close, because the builder moves teams, and the service outlives them.

⚠The reconstruction that fails silently

The subtle failure is a record that names versions nobody can dereference: corpus_version: 7 where version 7’s manifest was overwritten in place, extraction_run: 2026-03-02 where the extracted text was regenerated since. The stamp is only as good as the ability to open what it points to — manifests and eval results are append-only history, never files edited in place. The previous training said it of numbers; it is doubly true of the record that will be read in a dispute: reproducible or it did not happen.

Stop and escalate when a dependency turns out to have no version to stamp — most often the model, when a provider updates in place under a stable name. Record the fullest identifier the provider exposes, and route the residual to the service owner as a known reconstruction gap: it is theirs to accept in writing or to change providers over, not the builder’s to paper over with a guess.

KNOWLEDGE CHECK

A disputed answer from March carries all six version stamps. The reconstruction query runs - but corpus version 7's manifest was overwritten when version 8 shipped, and the March extraction text was regenerated since. What is the state of the investigation?

Key takeaway

Every answer gets six stamps — corpus, extraction, index, policy, prompt, model — because six dependencies move independently and any one changes the output; the reconstruction query turns a months-old dispute into 4.3’s routine diagnosis instead of an argument with a fluent artifact. The stamps are only as good as their dereference: manifests, extractions, and eval results are append-only history, never overwritten, or the record fails silently at the moment it is needed. The standard is the stranger test — a verdict reachable without the builder in the room. Lesson 6.2 lists what moves underneath you, and what each movement must trigger.

Search lessons