Drift and the Re-eval Triggers
Updated
IntermediateWhat you'll learn
~16 min- Name the six dependencies whose change invalidates standing eval results
- Wire each change to a proportionate re-evaluation, automatically where the change is detectable
- Catch the provider-side model update that arrives without asking
Prompt first: wire the triggers
Switch your workspace agent into plan mode and start with (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):
Plan re-evaluation triggers for our six stamped dependencies, pairing each change with a proportionate evaluation run and a scheduled byte-comparison canary for unannounced model drift, while appending every result to our evaluation history.Check that the plan or draft covers:
- Change detection per dependency: explicit manifest bumps for corpus changes, version-control diffs for extraction, index, policy, or prompt edits, and both provider announcements and canary drift for model changes.
- Proportionate re-evaluation tiers:
corpus bump: retrieval evaluation on affected classes plus the refusal set (since new documents can make old refusals answerable).re-extraction: retrieval evaluation on classes served by the re-extracted documents.index rebuild: retrieval evaluation across all classes.prompt or policy edit: full answer evaluation on affected classes, including the human hand-check sample.model change: the full evaluation battery across all classes.
- Scheduled byte-comparison canary: a small frozen probe set (including a refusal) compared byte-for-byte against the prior run to detect unannounced provider-side model updates.
- Append-only evaluation history: every triggered run recorded with the six dependency versions it graded, never overwriting prior records.
If the plan runs the full evaluation battery on every minor corpus bump or omits refusal re-testing on corpus growth, follow up with: "Scope corpus-bump triggers to affected-class retrieval plus the refusal set, and reserve full all-class answer evaluations for prompt, policy, or model changes."
Yesterday’s pass describes yesterday’s system
Module 4 ended with a carefully worded license: no failures observed, on this set, at this corpus version, with this configuration. This lesson is what those clauses cost: the moment any named thing changes, the license describes a system that no longer exists. Serving on last quarter’s eval after a corpus bump is not a small drift from rigor — it is exactly the unevaluated-class failure from 1.3, re-entering through the maintenance door.
The changes are not exotic. They are the system working as designed:
- The corpus moves — new filings land, a standard edition arrives, a document’s status flips. Each is a 2.4 version bump, each redraws what the system can and should say. The subtle one: new documents can make old refusal questions answerable — the refusal set must re-run on corpus growth, or the system keeps refusing questions it now could answer, and the eval keeps passing while it does.
- The policy moves — a resolution-rule edit, a serving-matrix change, a tier reassignment. Small diffs, answer-shaping consequences.
- The prompt contract moves — someone tightens a phrase after a 4.3 diagnosis. That is a change to the answering machine, and it re-earns its eval like any other.
- The model moves — and this one moves without asking.
The dependency that updates itself
Corpus, policy, and prompt changes announce themselves — they are your own commits and version bumps. The model is different: providers update models in place under stable names, and the answering behavior — the least stable layer in the whole system, as every module since 4.1 has had reason to note — shifts under you with no diff to review and no announcement you can rely on.
Hence the canary: a small fixed probe set — a handful of questions with stable expected behavior, including one refusal — run on a schedule, compared byte-for-byte against the last run. Prose drift on identical inputs with no local change is the tell that the model moved. The canary is deliberately cheap (it runs daily for pennies) and deliberately dumb (byte comparison, no judgment) — its job is not to grade the change but to detect it, so the real trigger — full re-eval — fires on evidence instead of on a provider changelog that may never come.
The failure mode of re-evaluation is not refusal — nobody argues against testing. It is cost-driven erosion: if every corpus bump demands the full battery, and corpus bumps are weekly, the re-eval quietly becomes quarterly, then ceremonial. The proportionate schedule in the prompt is what makes the discipline survivable: corpus changes re-run retrieval and refusals on affected classes; only the volatile layers — prompt, model — demand the full battery. 5.3 priced this; the design answer to “too expensive to run” is a cheaper proportionate tier, never a skipped one.
Stop and escalate when a re-eval fails after a provider-side model change — the service is now serving on a broken license, the rollback options (pin the prior model, block affected classes, drop to draft mode) all have costs, and choosing among them is the service owner’s call, made now, not after the weekend.
The canary has run daily for a month with identical output. This morning, three of its eight probes return differently worded answers - same claims, same citations, different prose. No corpus, policy, or prompt change shipped. What has the canary told you, and what fires?
Key takeaway
Every eval pass is licensed to a version set, and the license dies with the first change: corpus bumps redraw what the system should say — including making old refusals answerable, which is why the refusal set re-runs on growth; prompt and policy edits re-earn their evals like any other change to the machine; and the model updates itself, unannounced, which the daily byte-comparison canary converts from a silent drift into a detected event that fires the full battery. Proportionate tiers are what keep the discipline alive against its real enemy, cost erosion. Lesson 6.3 asks who answers for all of this — and when to turn it off.