intellimetrics Learning
Grounded Answers From Documents Module 3 · Retrieve the Evidence

The Build

Last reviewed

Intermediate

What you'll learn

~22 min
  • Build a working retrieval pipeline over the practice corpus with an AI CLI
  • Verify each stage's output before wiring the next, rather than trusting the pipeline end to end
  • Treat the tools as this month's choice and the contract as the permanent thing
⚠This lesson is dated on purpose

This is the only lesson in the training that names specific tools, and everything named here is a choice that was reasonable on this lesson’s review date — not the way to do it. The contract from 3.1 is the permanent thing; this lesson is one honest implementation of it, chosen because every tool is already on a stock machine: no accounts, no procurement, no admin rights. When the tools here age, the lesson gets re-stamped; the other twenty-one lessons will not need to change. That split is deliberate.

Setup, once

The practice corpus comes from one command — python3 scripts/t6-corpus/fetch.py from the training repository — which fetches the public documents, generates the field notes, and writes ./corpus with its manifest.json (Lesson 2.4 covers what is in it). It needs only python3 and pdftotext, and it tells you if pdftotext is missing.

Prompt first: the whole build

Build a retrieval pipeline over the practice corpus in ./corpus,
satisfying this contract: question in; passages out, each carrying
document id, revision, and page from the manifest; nothing outside
the corpus reaches any answer.


Implementation constraints - use only what is on this machine:
  - extraction: pdftotext (poppler), page boundaries preserved
  - index: SQLite FTS5 (ships inside Python's sqlite3) - lexical
    shape, per the 3.1 decision for this class
  - loop: a small Python script - index once, then query
  - provenance: every passage row stores (doc_id, revision, page)
    copied from manifest.json - never parsed out of the text


Build it in stages and STOP after each for my check:
  1. extract -> show me the per-document page counts vs the manifest
  2. index -> show me row counts and 3 sample rows with provenance
  3. query -> take a question, show the top 5 passages with scores
Do not wire a stage to the next until I have seen its output.

The staged stops are the discipline this site teaches everywhere: the agent builds, you verify each artifact before it becomes load-bearing. On a pipeline, that matters double — an extraction defect flows silently into the index and surfaces as a mysterious retrieval miss three stages later, where it costs ten times more to find.

What each check is for

Stage 1 — extraction counts. The manifest says FIST 3-30 (a public maintenance manual from the practice corpus) has 87 pages; the extraction produced 87 page records, or it did not. A shortfall is Lesson 2.2’s audit catching a real casualty before it enters the index. This check takes thirty seconds and is skipped by every tutorial ever written.

Stage 2 — provenance rows. Three sample rows, read by eye: does each passage carry a document id and page that exist in the manifest? The build’s most tempting shortcut is parsing provenance out of the extracted text (where the running headers lie about it) instead of carrying it from the manifest — this check catches that substitution while it is one line to fix.

Stage 3 — a real query. Ask a question you know the answer to, from Lesson 1.4’s build set. You are not grading answer quality yet — there is no answering. You are looking at passages: are they from the right document, is the score ordering sane, does the top passage actually bear on the question? This is your first contact with what retrieval actually returns, and it recalibrates everything you assumed.

Q: how often should transformer oil be sampled?


[1] 8.41  fist3-30 rev2018 p.41  "...oil samples should be taken
          annually as part of routine maintenance unless..."
[2] 7.92  fist4-1b rev2024 p.12  "...intervals in this volume
          supersede those in equipment-specific volumes..."
[3] 5.10  trnsfrmr rev2005 p.118 "...sampling frequency depends on
          criticality and loading..."

That is a passing stage 3: right documents, sane ordering, provenance on every line — and the corpus’s known conflict (2.1’s resolution rule) visible in the results, which is what “surface both” will look like downstream.

What is deliberately not here

No answering — the model that writes answers from these passages arrives in Module 4, after the eval harness that grades it. No semantic index — this class chose lexical in 3.1; the class that chose semantic follows the same staged build with an embedding step, and the checks do not change. No web access, no framework — frameworks bundle these stages precisely so you cannot check between them, which is the opposite of what a build you must later debug wants.

Stop and escalate when the build needs anything not already on the machine — an embedding service for a semantic class, a document store with an API. That is 3.1’s infrastructure conversation arriving on schedule: the platform team owns runtime services, and the class can usually ship its lexical baseline while that conversation happens.

KNOWLEDGE CHECK

The agent offers to build the full pipeline in one pass - 'extract, index, and query, tested end to end, in about a minute.' Why insist on the staged stops anyway?

Key takeaway

One honest implementation of the contract, from tools already on the machine: extract with page boundaries, index with provenance carried from the manifest, query and look at real passages — with a stop after every stage, because pipeline defects flow silently forward and are cheapest at the stage that made them. The tools are this lesson’s date; the contract and the staged checks are permanent. No answering yet, by design: the grader comes before the answerer. Lesson 3.3 puts the permission tier from 2.3 into the retrieval path, where it becomes real.

Search lessons