Extraction Is Where Corpora Die

Prompt first: audit the extraction

Extract text from every PDF in [corpus directory], one text file per
document, page boundaries preserved.

Then AUDIT your own extraction and report, per document:
  - pages yielding no text or near-none (likely scans)
  - pages where text density collapses (likely tables or figures)
  - repeated line fragments appearing on most pages (running
    headers/footers stitched into the text)
  - reading-order anomalies: sentences that end mid-thought at
    column or page boundaries

Rank documents by extraction quality. Do not summarize the corpus as
"successfully extracted" - the audit table IS the deliverable, and
"successfully extracted" is not a finding.

The audit is the lesson. Extraction always succeeds in the trivial sense — a text file comes out. Whether the text preserves what the document said is a separate question nobody asks, and it decides everything downstream: retrieval cannot find what extraction destroyed.

The four ways documents die on the way in

This problem is twenty years old, thoroughly unglamorous, and untouched by any model improvement — it happens before any model sees anything. On the practice corpus you will meet all four:

1. The scan with no text. A page that is a photograph of a page. Extraction yields nothing, or OCR yields an approximation with characteristic wounds — 0 for O, rn for m, table columns shuffled. The failure mode downstream is silence: the document is “in the corpus,” retrieval never returns it, and nobody knows the difference. Your refusal questions from Lesson 1.4 will pass for the wrong reason.

2. The flattened table. The transformer manual’s maintenance-interval table — the single most-asked-about content in the document — extracts as a vertical stream of cell fragments: Annually, Oil sample, 6 months, Visual. The association between row and column is gone, which means the fact is gone: “oil sample: annually” no longer exists as a retrievable statement. Tables hold exactly the facts people ask questions about; this is not an edge case, it is the center.

3. The stitched running header. FIST 3-30 TRANSFORMER MAINTENANCE 41 — FIST being the public maintenance-manual series in the practice corpus — appears mid-sentence on every extracted page, because the header sat between paragraphs in the layout. Retrieval now matches every page of the document for the query “transformer maintenance” with identical strength — the header outweighs the content.

4. Scrambled reading order. Two-column layouts read across instead of down; footnotes interleave into body text; a sentence ends and resumes three paragraphs later. The passage retrieval returns is word salad with a real citation.

Provenance survives, or the citation lies

Every extracted passage must carry: source document, document revision, page (or section), and extraction run. This is non-negotiable for one reason — the citation that Module 4’s hand check opens is built from these fields. Lose the page number in extraction and “FIST 3-30, p.41” becomes “FIST 3-30, somewhere,” and the ninety-second check becomes a forty-minute search, which means it stops being performed.

Practically: keep page boundaries in the extracted text (extractors emit form-feeds or per-page files), and record the extraction as part of the corpus manifest — Lesson 2.4 gives that a home.

What to do about the casualties

The audit gives you a ranked list of damage. Triage, in order of honesty:

1. Fix what matters. OCR the scans that serve a question class; re-extract tables that hold asked-about facts into a structure that preserves row-column association. Effort goes where questions are.
2. Mark what you did not fix. A document with known-bad extraction stays in the manifest flagged extraction: degraded — so when retrieval never returns it, the silence is documented rather than mysterious.
3. Exclude what you cannot account for. A document whose extraction is unusable is not in the corpus in any honest sense; carrying it as if it were answerable is a refusal question waiting to fail.

Stop and escalate when a document that matters to a question class resists extraction — a critical scanned filing, a manual that is all tables. Buying better OCR, re-requesting a digital original from the source, or rekeying ten load-bearing pages is a cost decision for the corpus owner, and “we quietly dropped it” is the one option not on the table.

Key takeaway

Extraction always produces a text file; whether it preserved the document is the question nobody asks, and the audit answers it: scans that yielded nothing, tables flattened into disassociated fragments, running headers stitched into every page, reading order scrambled. Tables hold exactly the facts people ask about, so this is the center, not an edge. Provenance — document, revision, page, extraction run — must survive or Module 4’s ninety-second check becomes a forty-minute search. Triage by question class, mark what you did not fix, exclude what you cannot account for. Lesson 2.3 decides who may see what the extraction preserved.