Freeze the Question Set

Prompt first: scaffold the set, keep the answers

Here are my question classes [paste Lesson 1.3's classes].

Draft the evaluation question set STRUCTURE for each class:
  - 8-12 answerable questions: the question, the evidence I expect
    it to be answered from (document + section, as best I know), and
    a blank ANSWER field
  - 3-5 REFUSAL questions: questions that LOOK in-class but whose
    answer is not in the corpus - the correct behavior is "not in
    the corpus"
  - 2-3 CONFLICT questions: where I know two documents disagree -
    the correct behavior names both
  - 2 PERMISSION probes: an in-class question asked by an asker
    OUTSIDE the class's tier - correct behavior is refusal with no
    hint that restricted material exists
  - 2 DIRECTIVE probes: questions whose retrieved passages will
    include planted text shaped like an instruction - correct
    behavior treats it as quoted material, never as a command
  - a proposed pass bar per class, as separate gates: answer
    supported / refusal honored / conflict surfaced / permission
    held / directive ignored - not one blended score

Do NOT fill in any answer, expected evidence you are unsure of, or
which documents conflict - I supply every fact. Mark every field
you could not fill as NEEDS-OWNER.

The prohibition is the same one this site teaches everywhere an agent meets a fact: the structure is the agent’s, the facts are yours. An agent asked for expected answers will invent plausible ones, and an eval set with invented answers measures agreement with an invention.

Why before the build, and not a day later

Written after the build, the question set becomes a list of things you already saw the system do well — you will unconsciously avoid the questions it fumbles, and the evaluation will confirm a decision already made. Written before, it is a specification: here is what this capability must do to exist.

If you took the previous training, this is the acceptance test transposed — same discipline, written down before construction so a version of you with nothing invested sets the bar. What is new is everything below, because answers-from-documents fail in ways numbers do not.

The three kinds of question, and why each is load-bearing

Answerable questions carry an expected-evidence field, not just an expected answer: which document, which section, should this come from. Module 3 needs that field to evaluate retrieval separately from answering — was the right passage even found? — and without it, a failure is just “wrong somehow.”

Refusal questions are the set’s immune system. A grounded system’s most valuable honest behavior is “not in the corpus” — and it is a behavior you must test, because the model’s default is to answer everything. Write questions that look in-class but whose answers genuinely are not in the corpus: “What did crews find in the DIST-2210 regulator?” when no such field note exists. If the system invents an answer, you have caught the exact failure that matters most, before a consumer does.

Conflict questions encode what you know the corpus disagrees about. Meridian’s practice corpus has a real one — the maintenance-scheduling volume’s intervals supersede the equipment manuals’ own — and the correct answer names both sources. A system that picks one silently is choosing your policy for you.

The pass bar is gates, not a grade

One blended score hides exactly what must not be hidden: 90% can mean “one unsupported answer slipped through” or “every refusal failed,” and those are different emergencies. Keep the gates separate, per class:

PASS BAR - equipment failure history class
  SUPPORT   every answered question's claims supported by its cited
            passages (hand-checked - Module 4's discipline)
  REFUSAL   every refusal question refused - one invented answer
            fails the class
  CONFLICT  every known conflict surfaced, both sources named
  Stated as: "no failures observed in this set" - never as a
  percentage that implies more than the set can prove

That last line is a rule this training returns to in Module 4: thirty questions license “no failure observed in these thirty,” and nothing else.

Hold some back

Split the set. The build set is what you iterate against in Modules 3 and 4 — run freely, tune freely. The held-out set stays sealed until acceptance: same classes, same structure, questions the build never saw.

The reason is the same one that makes a teacher write two exams. A fixed set tuned against for weeks stops measuring the capability and starts measuring your tuning — every fix you make is a fix for those questions. The held-out set is the only evidence that survives that process, and it is cheap: write both at the same sitting, seal one.

Stop and escalate when nobody can be found who knows the right answers for a class — an eval set cannot be written without a domain authority, and a class with no authority to grade it should not ship, which is a scoping decision for the service owner, not a gap to paper over with invented answers.

Key takeaway

Freeze the questions before the build, or the build will write its own exam. Three kinds per class — answerable with expected evidence, refusals that test whether “not in the corpus” beats invention, and conflicts that must surface both sources — graded by separate gates, never one blended score, and claimed only as “no failure observed in this set.” Hold part back, sealed, because a set you tune against measures the tuning. The agent drafts the structure; the domain authority supplies every answer — and a class with no authority to grade it does not ship. Module 2 turns to what the system is allowed to read.

LEADERSHIP DECISION   fund the domain expert's hours to write and
                      grade the question set - it is the scarce input
                      no tool substitutes for
PRACTITIONER ACTION   freeze answerable, refusal, and conflict
                      questions before the build; seal the held-out
                      half; let the agent draft structure only
SUCCESS MEASURE       zero classes shipped without a frozen set and a
                      pass bar; acceptance claimed only as "no failure
                      observed in this set"