intellimetrics Learning
Grounded Answers From Documents Module 2 · Govern What It Reads

Version the Corpus

Last reviewed

Intermediate

What you'll learn

~16 min
  • Assemble the corpus manifest as the single versioned record of what the system reads
  • State what a corpus version means and what changes it
  • Make the fetch itself reproducible, so the corpus can be rebuilt rather than remembered

The practice corpus, one command

For the training’s own exercises, the fetch script ships with the site’s repository: python3 scripts/t6-corpus/fetch.py fetches the public documents (U.S. government works, public domain), generates the Meridian field notes deterministically, extracts with page boundaries, and emits corpus/manifest.json — checksums pinned, so a source revising a document in place announces itself as exactly the corpus-version event this lesson describes. What follows is how to build the same artifact for your own corpus.

Prompt first: assemble the manifest

Assemble the corpus manifest from what Module 2 produced: the
selection list with statuses (2.1), the extraction audit (2.2), and
the permission map (2.3).


One entry per document:
  id, title, source (URL or system), source revision or edition,
  fetch date, checksum, class(es) served, status (authoritative /
  superseded / reference), access tier, extraction quality
  (clean / degraded / excluded), extraction run id


Plus a header: corpus version (start at 1), date, owner, and the
resolution rule from 2.1 by reference.


Then write the FETCH SCRIPT that reproduces this corpus from
sources: fetch, checksum-verify, extract, emit the manifest. Flag
any document that cannot be re-fetched from its source - those are
the ones a rebuild would lose.

The manifest is Module 2’s closing artifact, and everything in it already exists — this lesson is assembly plus one new idea: the version.

What a corpus version means

One number, incremented when anything about what-the-system-reads changes: a document added or removed, a status flipped (the new standard edition arrives; last year’s becomes superseded), a tier changed, a re-extraction. The rule for what forces the increment is simple — anything that could change an answer.

Why a version and not just a changelog: three later artifacts point at it, and all three are only as good as the pointer.

  • Every evaluation (Module 4) records the corpus version it graded. “The system passed” is meaningless without “against corpus v3” — a document added since is un-evaluated territory.
  • Every answer (Module 6) records the corpus version that produced it. When the supervisor asks in March why January’s answer said something different, the answer record plus two manifest versions is the entire investigation.
  • Every re-eval trigger (Module 6) is defined as: the version moved.

If you took Building Trustworthy Data Products, this is its run-record discipline arriving one module earlier than it did there — because a corpus changes for external reasons (new filings arrive, standards get revised) even when nobody touches the system, so the versioning cannot wait for the operations module.

The fetch is part of the version

A corpus you cannot rebuild is a corpus you can only remember. The fetch script — fetch from source, verify checksum, extract, emit manifest — makes corpus v1 a procedure rather than a pile: run it again and you get v1 again, or a named list of what changed upstream.

Two honest cases from the practice corpus:

  • Re-fetchable: the public manuals and standards live at stable URLs; the script fetches, checksums confirm nothing moved. A rebuild is a command.
  • Not re-fetchable: documents that arrived by email, an export someone ran once, the filing downloaded from a portal that has since reorganized. These go in the manifest flagged archival-copy-only — the corpus keeps the copy precisely because the source cannot be trusted to.
💡The checksum earns its place the first time it fails

A checksum mismatch on re-fetch means the source changed the document without changing its name — a revised manual at the same URL, a corrected filing. That is not an error to suppress; it is the corpus-version increment announcing itself. The script that hides this behind a silent re-download has converted your versioned corpus back into a pile.

Stop and escalate when a document that serves a question class exists only as someone’s local copy with no authoritative source — whether the organization adopts it as a record (giving it a home and an owner) or the class does without it is a records-management decision, and the manifest flags it rather than settles it.

KNOWLEDGE CHECK

A new edition of the construction standard is fetched into the corpus and the old one is marked superseded. Nobody increments the corpus version. What breaks, and when?

Key takeaway

The manifest assembles Module 2 into one versioned record — per document: source, revision, checksum, classes, status, tier, extraction quality — and the version increments on anything that could change an answer. Three things point at it: every evaluation, every answer record, every re-eval trigger, and each is only as good as the pointer. The fetch script makes the corpus a procedure instead of a pile, checksums announce upstream changes instead of hiding them, and un-re-fetchable documents get flagged rather than forgotten. Module 3 finally builds the thing that reads all of this.

Search lessons