Grounded Answers From Documents Module 3 · Retrieve the Evidence

The Build

Updated

Intermediate

What you'll learn

~22 min
  • Build a working retrieval pipeline over the practice corpus with an AI CLI
  • Verify each stage's output before wiring the next, rather than trusting the pipeline end to end
  • Treat the tools as this month's choice and the contract as the permanent thing
⚠This lesson is dated on purpose

This is the only lesson in the training that names specific tools, and everything named here is a choice that was reasonable on this lesson’s review date — not the way to do it. The contract from 3.1 is the permanent thing; this lesson is one honest implementation of it, chosen because every tool is already on a stock machine: no accounts, no procurement, no admin rights. When the tools here age, the lesson gets re-stamped; the other twenty-one lessons will not need to change. That split is deliberate.

Setup, once

The practice corpus comes from one command — python3 scripts/t6-corpus/fetch.py from the training repository — which fetches the public documents, generates the field notes, and writes ./corpus with its manifest.json (Lesson 2.4 covers what is in it). It needs only python3 and pdftotext, and it tells you if pdftotext is missing.

Prompt first: the whole build

Switch your workspace agent into plan mode on the repository directory for ./corpus and start with (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):

Plan a three-stage local retrieval pipeline over ./corpus using pdftotext and SQLite FTS5 that carries document id, revision, and page from manifest.json, stopping after each stage so I can verify its output first.

Check that the plan or draft covers:

  • Local stock tools only: extraction with pdftotext (poppler) preserving page boundaries, and a lexical index in SQLite FTS5 via Python’s built-in sqlite3.
  • Manifest-carried provenance: every indexed passage row storing (doc_id, revision, page) copied directly from manifest.json, never parsed out of running headers or body text.
  • Stage 1 stop (Extraction): comparing per-document extracted page counts against manifest.json before indexing.
  • Stage 2 stop (Indexing): showing total row counts and three sample rows with full provenance before querying.
  • Stage 3 stop (Querying): taking a test question and returning the top five ranked passages with scores and provenance, without wiring an answering model yet.

If the agent tries to build and run all three stages in one uninterrupted pass, follow up with: "Stop after Stage 1 extraction and show me the per-document page counts compared against manifest.json before writing the index step."

The staged stops are the discipline this site teaches everywhere: the agent builds, you verify each artifact before it becomes load-bearing. On a pipeline, that matters double — an extraction defect flows silently into the index and surfaces as a mysterious retrieval miss three stages later, where it costs ten times more to find.

What each check is for

Stage 1 — extraction counts. The manifest says FIST 3-30 (a public maintenance manual from the practice corpus) has 87 pages; the extraction produced 87 page records, or it did not. A shortfall is Lesson 2.2’s audit catching a real casualty before it enters the index. This check takes thirty seconds and is skipped by every tutorial ever written.

Stage 2 — provenance rows. Three sample rows, read by eye: does each passage carry a document id and page that exist in the manifest? The build’s most tempting shortcut is parsing provenance out of the extracted text (where the running headers lie about it) instead of carrying it from the manifest — this check catches that substitution while it is one line to fix.

Stage 3 — a real query. Ask a question you know the answer to, from Lesson 1.4’s build set. You are not grading answer quality yet — there is no answering. You are looking at passages: are they from the right document, is the score ordering sane, does the top passage actually bear on the question? This is your first contact with what retrieval actually returns, and it recalibrates everything you assumed.

Q: how often should transformer oil be sampled?
[1] 8.41 fist3-30 rev2018 p.41 "...oil samples should be taken
annually as part of routine maintenance unless..."
[2] 7.92 fist4-1b rev2024 p.12 "...intervals in this volume
supersede those in equipment-specific volumes..."
[3] 5.10 trnsfrmr rev2005 p.118 "...sampling frequency depends on
criticality and loading..."

That is a passing stage 3: right documents, sane ordering, provenance on every line — and the corpus’s known conflict (2.1’s resolution rule) visible in the results, which is what “surface both” will look like downstream.

What is deliberately not here

No answering — the model that writes answers from these passages arrives in Module 4, after the eval harness that grades it. No semantic index — this class chose lexical in 3.1; the class that chose semantic follows the same staged build with an embedding step, and the checks do not change. No web access, no framework — frameworks bundle these stages precisely so you cannot check between them, which is the opposite of what a build you must later debug wants.

Stop and escalate when the build needs anything not already on the machine — an embedding service for a semantic class, a document store with an API. That is 3.1’s infrastructure conversation arriving on schedule: the platform team owns runtime services, and the class can usually ship its lexical baseline while that conversation happens.

KNOWLEDGE CHECK

The agent offers to build the full pipeline in one pass - 'extract, index, and query, tested end to end, in about a minute.' Why insist on the staged stops anyway?

Key takeaway

One honest implementation of the contract, from tools already on the machine: extract with page boundaries, index with provenance carried from the manifest, query and look at real passages — with a stop after every stage, because pipeline defects flow silently forward and are cheapest at the stage that made them. The tools are this lesson’s date; the contract and the staged checks are permanent. No answering yet, by design: the grader comes before the answerer. Lesson 3.3 puts the permission tier from 2.3 into the retrieval path, where it becomes real.

Search lessons