Grounded Answers From Documents Module 3 · Retrieve the Evidence

Did It Find the Evidence?

Updated

Intermediate

What you'll learn

~18 min
  • Evaluate retrieval separately from answering, using the expected-evidence fields from the question set
  • Diagnose a retrieval miss to a specific cause rather than a general shrug
  • Know the fixes that belong to the corpus versus the ones that belong to the retriever

Prompt first: grade the retrieval alone

Switch your workspace agent into plan mode and start with (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):

Using the answerable questions and expected-evidence fields from our build set, plan a retrieval-only evaluation that checks candidate ranks without any answering model and groups every miss by shared pattern. Ask if anything is unclear.

Check that the plan or draft covers:

  • Retrieval-only execution: running each answerable build-set question through retrieval alone, with no answering model invoked.
  • Rank and displacement reporting: recording whether a passage from the expected document and section appears in the top ten candidates, its exact rank (or MISS), and what ranked above it when it landed low.
  • Diagnostic table and grouped misses: one summary table (question, expected evidence, found-at-rank or MISS, top-ranked passage), followed by misses grouped by shared traits (same document, phrasing pattern, or section type) rather than one-off speculation.

If the agent speculates on individual misses instead of grouping them, follow up with: "Group the retrieval misses by shared document, section type, or phrasing mismatch so we can separate extraction casualties from vocabulary gaps."

This eval can run today because Module 1 made it possible: every answerable question carries an expected-evidence field. Without that field, retrieval evaluation is impossible and every failure is just “wrong somehow” — which is exactly why the field was mandatory.

Why retrieval gets graded before any answer exists

An answer can only be as good as the evidence the model was handed. When the final answer is wrong, the first question is always was the right passage even in what the model saw? — and if you cannot answer that, you cannot fix anything: you will tune prompts to repair a corpus problem, or re-extract documents to repair a prompt problem.

Grading retrieval alone, now, gives you the baseline that makes Module 4’s split diagnosis possible. It also finds an entire failure category while it is still cheap: the miss.

The anatomy of a miss

The expected passage exists in the corpus and did not come back. Every miss has a cause in one of four places, and the grouped-miss table from the prompt is how you find which:

1. It was never extracted. The passage lives on a scanned page or inside a flattened table — Lesson 2.2’s audit predicted this miss before it happened. Check the extraction audit first, always: a retrieval cannot find text that does not exist in the index. Fix: corpus (re-extraction), never the retriever.

2. It was extracted and mangled. The text is present but the running header stitched through it, or the table fragments scattered — the words are there, the statement is not. Fix: corpus (extraction cleanup).

3. Vocabulary gap. The passage says “arcing damage on the contact stack”; the question says “burnt contacts.” Lexical retrieval has no bridge. This is 3.1’s shape decision showing up as data — a class with real vocabulary-gap misses has just demonstrated the specific evidence that justifies a semantic or hybrid shape. Fix: retriever (shape change), and now with a named failing question, which is what 3.1 demanded before allowing the upgrade.

4. Drowned. The passage came back at rank 40 under a pile of passages that mention the terms more often while bearing on the question less. Often the running-header pollution from 2.2, sometimes genuine ambiguity in the question. Fix: split — header pollution is corpus; ambiguity is a question-class boundary problem from 1.3.

💡The grouped misses are the real report

One miss is an anecdote. The grouping is the diagnosis: all misses in one document → extraction casualty; all misses on one phrasing → vocabulary gap; misses spread evenly → look at the index itself. The prompt demands groups because per-miss speculation produces twenty plausible stories, and the grouping produces one actionable one.

What a passing retrieval eval licenses — and what it does not

Expected evidence in the top handful of candidates for every answerable question, misses triaged to owners: that is the gate to Module 4. It licenses exactly one claim — when the model answers, the right evidence was in front of it.

It does not license “the answers will be right.” The model can misuse evidence it was correctly handed — that is the other half, and it fails independently. Keeping the two evals separate is what makes “which half failed” answerable at all, and that question is Module 4’s hinge.

Stop and escalate when the misses concentrate in a document the corpus owner marked load-bearing and extraction cannot recover it — that is 2.2’s rekeying-cost decision arriving with data attached: here are the questions that fail, here is the document that fails them, the spend decision is the owner’s.

KNOWLEDGE CHECK

A question's expected passage exists in the corpus - you can open the PDF and read it - but it never appears in the top 50 retrieval candidates. What do you check first?

Key takeaway

Grade retrieval alone, before any answer exists, against the expected-evidence fields the question set carried for exactly this moment — because when a final answer is wrong, everything depends on knowing whether the right passage was ever in front of the model. Every miss has a home: never extracted, extracted-and-mangled, vocabulary gap, or drowned — the first two are corpus fixes, the third is the named evidence a shape upgrade requires, and the grouping, not the anecdote, is the diagnosis. A passing eval licenses one claim only: the evidence was there. Whether the answer honors it is Module 4’s question.

Search lessons