intellimetrics Learning
Grounded Answers From Documents Module 3 · Retrieve the Evidence

Did It Find the Evidence?

Last reviewed

Intermediate

What you'll learn

~18 min
  • Evaluate retrieval separately from answering, using the expected-evidence fields from the question set
  • Diagnose a retrieval miss to a specific cause rather than a general shrug
  • Know the fixes that belong to the corpus versus the ones that belong to the retriever

Prompt first: grade the retrieval alone

Run the retrieval eval - no answering model anywhere in this.


For every ANSWERABLE question in the build set [paste], the question
set names expected evidence (document + section). For each:
  - run the question through retrieval as built
  - report whether a passage from the expected evidence appears in
    the top 10 candidates, and at what rank
  - report what ranked ABOVE it when it appears low


Output one table: question, expected evidence, found-at-rank (or
MISS), top-ranked passage. Then group the misses by what they have
in common - same document? same question phrasing? same section
type? - and give me the groups, not per-miss speculation.

This eval can run today because Module 1 made it possible: every answerable question carries an expected-evidence field. Without that field, retrieval evaluation is impossible and every failure is just “wrong somehow” — which is exactly why the field was mandatory.

Why retrieval gets graded before any answer exists

An answer can only be as good as the evidence the model was handed. When the final answer is wrong, the first question is always was the right passage even in what the model saw? — and if you cannot answer that, you cannot fix anything: you will tune prompts to repair a corpus problem, or re-extract documents to repair a prompt problem.

Grading retrieval alone, now, gives you the baseline that makes Module 4’s split diagnosis possible. It also finds an entire failure category while it is still cheap: the miss.

The anatomy of a miss

The expected passage exists in the corpus and did not come back. Every miss has a cause in one of four places, and the grouped-miss table from the prompt is how you find which:

1. It was never extracted. The passage lives on a scanned page or inside a flattened table — Lesson 2.2’s audit predicted this miss before it happened. Check the extraction audit first, always: a retrieval cannot find text that does not exist in the index. Fix: corpus (re-extraction), never the retriever.

2. It was extracted and mangled. The text is present but the running header stitched through it, or the table fragments scattered — the words are there, the statement is not. Fix: corpus (extraction cleanup).

3. Vocabulary gap. The passage says “arcing damage on the contact stack”; the question says “burnt contacts.” Lexical retrieval has no bridge. This is 3.1’s shape decision showing up as data — a class with real vocabulary-gap misses has just demonstrated the specific evidence that justifies a semantic or hybrid shape. Fix: retriever (shape change), and now with a named failing question, which is what 3.1 demanded before allowing the upgrade.

4. Drowned. The passage came back at rank 40 under a pile of passages that mention the terms more often while bearing on the question less. Often the running-header pollution from 2.2, sometimes genuine ambiguity in the question. Fix: split — header pollution is corpus; ambiguity is a question-class boundary problem from 1.3.

💡The grouped misses are the real report

One miss is an anecdote. The grouping is the diagnosis: all misses in one document → extraction casualty; all misses on one phrasing → vocabulary gap; misses spread evenly → look at the index itself. The prompt demands groups because per-miss speculation produces twenty plausible stories, and the grouping produces one actionable one.

What a passing retrieval eval licenses — and what it does not

Expected evidence in the top handful of candidates for every answerable question, misses triaged to owners: that is the gate to Module 4. It licenses exactly one claim — when the model answers, the right evidence was in front of it.

It does not license “the answers will be right.” The model can misuse evidence it was correctly handed — that is the other half, and it fails independently. Keeping the two evals separate is what makes “which half failed” answerable at all, and that question is Module 4’s hinge.

Stop and escalate when the misses concentrate in a document the corpus owner marked load-bearing and extraction cannot recover it — that is 2.2’s rekeying-cost decision arriving with data attached: here are the questions that fail, here is the document that fails them, the spend decision is the owner’s.

KNOWLEDGE CHECK

A question's expected passage exists in the corpus - you can open the PDF and read it - but it never appears in the top 50 retrieval candidates. What do you check first?

Key takeaway

Grade retrieval alone, before any answer exists, against the expected-evidence fields the question set carried for exactly this moment — because when a final answer is wrong, everything depends on knowing whether the right passage was ever in front of the model. Every miss has a home: never extracted, extracted-and-mangled, vocabulary gap, or drowned — the first two are corpus fixes, the third is the named evidence a shape upgrade requires, and the grouping, not the anecdote, is the diagnosis. A passing eval licenses one claim only: the evidence was there. Whether the answer honors it is Module 4’s question.

Search lessons