The Kill Gate

Prompt first: run the gate

Before anything else in this training, this prompt — pointed at your own candidate questions:

Here are the questions people actually asked that made us consider
building document QA: [paste them, verbatim, as asked]

For EACH question, route it through four exits in order and stop at
the first that works:
  1. READ IT - is the answer in one known document under ~30 pages?
     Name the document.
  2. SEARCH BOX - would keyword search over the document titles and
     text find it in under five minutes? Say what the search term
     would be.
  3. GOVERNED TABLE - is this a how-many/which-ones question that a
     structured data product answers or should answer? Name the table
     that would.
  4. JUDGMENT - does answering require weighing, deciding, or
     predicting rather than locating? Say so.

Only questions that pass ALL FOUR exits remain candidates for a
grounded QA build. Do not soften the routing to keep candidates
alive - killing a candidate here is the cheap outcome.

The rest of this lesson explains why each exit exists and what it costs to skip the gate.

The gate, exit by exit

Exit 1 — read the document. “What does the transformer manual say about oil sampling intervals?” has an answer on a known page of a known manual. Opening it takes four minutes. A build that answers it takes weeks and returns a paraphrase of the page you should have opened. If the answer lives in one identifiable document of readable length, the method is reading.

Exit 2 — a search box. Keyword search over filenames and full text is forty years old, ubiquitous, and dramatically underused. “Which filings mention the Maple Creek substation?” is a search, not a question — the deliverable is a list of documents, and search boxes produce exactly that. If a five-minute search answers it, the method is search.

Exit 3 — a governed table. “How many circuits had more than three fault events last quarter?” arrived phrased as a document question because the asker did not know a table existed. It is a structured query, and if you took the previous training you know what a governed answer to it looks like — owned definition, reconciled, rerunnable. Route it there. A document system answering a counting question is the wrong tool succeeding badly.

Exit 4 — judgment. “Should we replace the regulator bank or keep repairing it?” requires weighing cost, risk, and history. No retrieval answers it. Retrieval can feed it — the maintenance history, the manual’s guidance — but the answer is a decision someone makes, and a system that pretends otherwise is manufacturing authority it does not have.

What survives all four exits is a specific shape: a locating question whose answer is spread across many documents, or buried in documents nobody can name. “What did crews find inside regulator banks of this type, across ten years of field notes?” — no single document to read, no keyword that finds it (the notes say “burnt contact stack,” not “regulator failure mode”), no table that holds it, no judgment required. That is the question this training builds for.

Run it per class, not per project

The gate is not a one-time decision about whether to build. It runs on every class of question — Lesson 1.3 defines classes properly — because the honest answer is usually mixed: the field-note questions survive, the manual-lookup questions exit at 1, the counting questions exit at 3. A project-level yes/no flattens that mix into either an overbuilt system answering questions a search box owns, or no system and the surviving questions still unanswered.

Keep the routed list. It becomes Module 1’s first artifact: the questions that survived, and — just as valuable — the documented reasons the others exited, so the next person who proposes “let’s build a chatbot over everything” starts from your routing instead of from enthusiasm.

Stop and escalate when a question keeps being re-proposed after the gate killed it — that is not a routing problem, it is a stakeholder who wants the build for its own sake, and the decision belongs to whoever owns the budget, made with your routed list in front of them.

Key takeaway

Four exits, in order, per question class: read it, search it, query it, or admit it needs judgment — and only what survives all four is a candidate for the build. The gate’s value is exactly the candidates it kills: each one is a corpus you did not govern and an evaluation you did not maintain. What survives has a specific shape — locating questions whose answers are spread across documents nobody can name — and Lesson 1.3 turns those survivors into bounded classes a service can actually commit to.