intellimetrics Learning
Grounded Answers From Documents Module 6 · Re-test Every Changed Dependency

Drift and the Re-eval Triggers

Last reviewed

Intermediate

What you'll learn

~16 min
  • Name the six dependencies whose change invalidates standing eval results
  • Wire each change to a proportionate re-evaluation, automatically where the change is detectable
  • Catch the provider-side model update that arrives without asking

Prompt first: wire the triggers

Wire re-evaluation triggers for the QA service. The six stamped
dependencies (6.1) are the trigger list - for each, tell me:


  1. HOW the change is detected: corpus version bump (explicit),
     policy/prompt edits (version control), model change (provider
     announcement AND the canary below - providers update in place)
  2. WHAT re-runs, proportionate to the change:
       corpus bump      -> retrieval eval on affected classes + the
                           refusal set (new documents can make old
                           refusals answerable!)
       re-extraction    -> retrieval eval on the re-extracted
                           documents' classes - extraction changes
                           what is findable (2.2)
       index rebuild    -> retrieval eval, all classes - ranking
                           moved even if nothing else did
       prompt/policy    -> full answer eval on affected classes,
                           hand-check sample included
       model change     -> everything, all classes - the answering
                           behavior is the least stable layer
  3. The CANARY: a small fixed probe set run on a schedule, whose
     answers are compared byte-for-byte against last run - drift
     with no announced change is itself the alarm


Every trigger firing appends to the eval history with the versions
it graded (6.1's append-only rule). Nothing overwrites.

Yesterday’s pass describes yesterday’s system

Module 4 ended with a carefully worded license: no failures observed, on this set, at this corpus version, with this configuration. This lesson is what those clauses cost: the moment any named thing changes, the license describes a system that no longer exists. Serving on last quarter’s eval after a corpus bump is not a small drift from rigor — it is exactly the unevaluated-class failure from 1.3, re-entering through the maintenance door.

The changes are not exotic. They are the system working as designed:

  • The corpus moves — new filings land, a standard edition arrives, a document’s status flips. Each is a 2.4 version bump, each redraws what the system can and should say. The subtle one: new documents can make old refusal questions answerable — the refusal set must re-run on corpus growth, or the system keeps refusing questions it now could answer, and the eval keeps passing while it does.
  • The policy moves — a resolution-rule edit, a serving-matrix change, a tier reassignment. Small diffs, answer-shaping consequences.
  • The prompt contract moves — someone tightens a phrase after a 4.3 diagnosis. That is a change to the answering machine, and it re-earns its eval like any other.
  • The model moves — and this one moves without asking.

The dependency that updates itself

Corpus, policy, and prompt changes announce themselves — they are your own commits and version bumps. The model is different: providers update models in place under stable names, and the answering behavior — the least stable layer in the whole system, as every module since 4.1 has had reason to note — shifts under you with no diff to review and no announcement you can rely on.

Hence the canary: a small fixed probe set — a handful of questions with stable expected behavior, including one refusal — run on a schedule, compared byte-for-byte against the last run. Prose drift on identical inputs with no local change is the tell that the model moved. The canary is deliberately cheap (it runs daily for pennies) and deliberately dumb (byte comparison, no judgment) — its job is not to grade the change but to detect it, so the real trigger — full re-eval — fires on evidence instead of on a provider changelog that may never come.

⚠Proportionate, or it will be skipped

The failure mode of re-evaluation is not refusal — nobody argues against testing. It is cost-driven erosion: if every corpus bump demands the full battery, and corpus bumps are weekly, the re-eval quietly becomes quarterly, then ceremonial. The proportionate schedule in the prompt is what makes the discipline survivable: corpus changes re-run retrieval and refusals on affected classes; only the volatile layers — prompt, model — demand the full battery. 5.3 priced this; the design answer to “too expensive to run” is a cheaper proportionate tier, never a skipped one.

Stop and escalate when a re-eval fails after a provider-side model change — the service is now serving on a broken license, the rollback options (pin the prior model, block affected classes, drop to draft mode) all have costs, and choosing among them is the service owner’s call, made now, not after the weekend.

KNOWLEDGE CHECK

The canary has run daily for a month with identical output. This morning, three of its eight probes return differently worded answers - same claims, same citations, different prose. No corpus, policy, or prompt change shipped. What has the canary told you, and what fires?

Key takeaway

Every eval pass is licensed to a version set, and the license dies with the first change: corpus bumps redraw what the system should say — including making old refusals answerable, which is why the refusal set re-runs on growth; prompt and policy edits re-earn their evals like any other change to the machine; and the model updates itself, unannounced, which the daily byte-comparison canary converts from a silent drift into a detected event that fires the full battery. Proportionate tiers are what keep the discipline alive against its real enemy, cost erosion. Lesson 6.3 asks who answers for all of this — and when to turn it off.

Search lessons