Which Half Failed
Last reviewed
AdvancedWhat you'll learn
~20 min- Split every answer failure into its retrieval half or its generation half before proposing any fix
- Route each failure type to its owner using the diagnostic evidence both evals already produce
- Refuse the reflex fix - prompt-tuning a corpus problem or re-indexing a discipline problem
What it is: the diagnostic split for every wrong answer — was the right evidence never found, or was found evidence misused — with a routing table naming who owns each fix.
What it buys: eval failures become work orders instead of arguments. Without the split, every wrong answer starts a debate about “the AI being wrong”; with it, a failure lands on the corpus owner, the retrieval builder, or the answering configuration in minutes, with evidence attached.
What to fund: nothing — the two evals already produce the diagnostic evidence. This is the discipline of reading them in order.
One wrong answer, two possible machines
Every wrong answer came from one of two failures, and they share nothing — not cause, not fix, not owner:
The retrieval half failed. The passage that would have supported a right answer never reached the model. The model then did the best it could with wrong ingredients — often fluently. No prompt improves this; the model cannot cite what it never saw. The fix lives in Module 2 or 3: extraction, corpus coverage, retrieval shape.
The generation half failed. The right passage was in front of the model, and the answer misstated it, ignored it for a worse passage, or blended it into something neither source says. The corpus is innocent. The fix lives in the answering: the prompt contract, abstention discipline, or the model itself.
The diagnostic is mechanical, because Module 3 and 4’s evals were designed to make it so:
For each failed answer: 1. Look at what retrieval returned (it was logged with the answer). 2. Was the expected evidence - 1.4's field - in the candidates? NO -> retrieval half. Go to 3.4's miss taxonomy: never-extracted / mangled / vocabulary gap / drowned. YES -> generation half. Go to 4.1's grades: adjacent claim / dropped qualifier / wrong-passage preference / hedge instead of refusal.Two questions, both answerable from logs that already exist. That is the entire method.
The routing table
| Failure signature | Half | Owner | The fix that actually fixes it |
|---|---|---|---|
| Expected passage absent from index | retrieval | corpus owner | re-extraction or acquisition (2.2’s triage) |
| Passage indexed but mangled | retrieval | corpus owner | extraction cleanup, re-index |
| Vocabulary gap, named failing question | retrieval | retrieval builder | shape upgrade per 3.1’s rule |
| Passage at rank 40 under noise | retrieval | corpus owner first | header cleanup before any ranking work |
| Adjacent claims, dropped qualifiers | generation | answering config | tighten the claim-per-citation contract |
| Right passage present, worse one cited | generation | answering config | passage selection in the prompt contract |
| Hedged instead of refused | generation | answering config | abstention instruction, re-run 4.2’s refusals |
| Correct grounding, obsolete source | neither | corpus owner | status/resolution rule (2.1) — the answer honored a corpus defect |
The last row is the one that keeps teams honest. An answer faithfully citing the superseded 2010 standard is both halves working perfectly — the failure is governance, and no amount of retrieval or prompt work touches it.
When an answer is wrong, the visible artifact is text, so the reflex is to fix the text-maker: tune the prompt, admonish the model, add instructions. But retrieval failures are the majority in young systems — extraction casualties and coverage gaps — and every hour spent prompt-tuning a retrieval failure makes the system worse: the prompt accretes special-case instructions compensating for evidence that was never there, and those instructions distort the answers whose evidence was fine.
The discipline is the order: retrieval question first, always. The generation half may only be blamed after the expected evidence is confirmed present in the candidates.
What this makes possible
The split converts Module 4’s eval results from a score into a work queue. “The class failed at 82%” starts an argument. “Three misses from the scanned filing (corpus owner, re-key decision pending), two adjacent-claim answers (contract tightened, re-run scheduled), one obsolete-source answer (status fix, resolution rule)” is a Tuesday.
It also feeds every stop-and-escalate this training has issued: the routing table is where those escalations get their address. The corpus owner receives extraction casualties with the failing questions attached; the platform conversation receives the shape upgrade with its named evidence; the service owner receives the invented answer with its graded citations. Nobody receives “the AI was wrong.”
Stop and escalate when a failure resists the split — expected evidence present, claims graded supported, answer still wrong for the asker’s purpose. That is usually a question-class boundary defect (1.3) or a corpus governance gap (2.1) wearing an eval failure’s clothes, and it goes to the service owner as a design question, not to either half as a bug.
An answer about sampling intervals is wrong. Retrieval logs show the correct passage was candidate #2. The answer cited candidate #7, a superseded manual, and stated its interval accurately. Which half failed, and what is the fix?
Key takeaway
Every wrong answer is one of two failures that share nothing: evidence never found (fix in the corpus or retriever) or evidence misused (fix in the answering) — plus the humbling third row where both halves worked and the corpus itself was wrong. The diagnostic is two questions against logs that already exist, retrieval first, always, because the reflex fix — prompting harder — actively damages a system whose real failure was upstream. The routing table turns eval scores into owned work orders and gives every escalation in this training its address. Lesson 4.4 states, plainly, what all of this machinery can and cannot promise.
LEADERSHIP DECISION require every "the AI was wrong" report to arrive routed - which half, which owner - before any fix is fundedPRACTITIONER ACTION retrieval question first, always; route by the table; never prompt-tune a miss whose evidence was absentSUCCESS MEASURE eval failures become owned work orders in minutes; zero prompt changes shipped to repair corpus defects