Grounded Answers From Documents Module 3 · Retrieve the Evidence

Retrieval Respects Permissions

Updated

Advanced

What you'll learn

~18 min
  • Enforce access tiers inside retrieval, before any passage reaches the model
  • Explain why filtering the answer instead of the evidence always fails
  • Close the citation loophole: opening a source re-checks access
ℹLeadership brief

What it is: the access tiers from corpus design, enforced at the exact point where passages are fetched — before the model ever sees them — with citations re-checking access when opened.

What it buys: the oversharing failure becomes structurally impossible instead of policy-discouraged. A model cannot leak a passage it never received, which is a stronger guarantee than any instruction to a model can be.

What to fund: nothing new — this is a filter clause in the retrieval built last lesson. The cost is design attention, and the alternative was priced in 2.3’s brief: re-indexing under incident pressure.

Prompt first: put the tier in the query path

Switch your workspace agent into plan mode on your retrieval workspace and start with (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):

Plan how to extend our retrieval pipeline so access tiers from the manifest filter candidates inside the search query itself, re-check access when opening citations, and audit any code path that bypasses the filter.

Check that the plan or draft covers:

  • Candidate-level filtering: excluding passages whose document tier exceeds the asker’s context inside the retrieval query itself, never post-filtering results or relying on the model.
  • Silent exclusion: queries that drop restricted passages behaving identically to queries where those documents never existed, with no “results withheld” hint.
  • Citation open re-check: opening a cited source taking the opener’s context and checking the document’s tier against the current manifest.json before serving.
  • Side-by-side tier test and bypass audit: running the same question as all-staff and as contracts-manager side by side, plus listing any debug, eval, or logging path where passages bypass the filter.

If the plan filters results after ranking or forgets citation re-checks, follow up with: "Move the tier check into the candidate search query itself, add a current-manifest check on citation open, and list every bypass path."

Why the filter must live in retrieval, not after it

There are three places people put this control, and two of them fail:

Filtering the answer — let the model see everything, instruct it not to reveal restricted content. Fails structurally: the model’s answer is derived from what it read, and derivation leaks. A summary shaped by the board minute reveals the minute without quoting it, and no instruction reliably prevents what the site’s own security training states as the ground rule — content the model reads is content the model may act on. Instructions are not a boundary.

Filtering the results — retrieve everything, drop restricted passages before the model sees them. Nearly right, and subtly wrong twice: the restricted passages still shaped the ranking (their presence displaces what the asker should have seen), and every code path that touches results-before-filtering is one refactor away from skipping the filter.

Filtering the candidates — the search itself runs over only what the asker may see. The restricted passage is not late-dropped; it was never a candidate. A model cannot leak what it never received — a structural guarantee for the passage-content channel, indifferent to model behavior, prompt injection, or instruction-following. It is the strongest single control in this training, and it governs one channel: what 5.1 adds (aggregation) and the silence rule below (presence) are the channels it does not cover.

The silence requirement

Requirement 2 in the prompt deserves its own paragraph, because it is a real design tension. When the filter removes passages, should the asker be told “some results were withheld”?

No — and 2.3’s map explains why. The existence of a restricted document bearing on this question is itself information at the restricted tier: “results withheld” on a question about a pending acquisition tells the analyst what the board is discussing. The honest behaviors are the ones from 1.4’s eval set: answer from what is visible, or refuse as “not in the corpus” — from this asker’s view, it is not.

The citation loophole

The filter guards question-time. Citations create a second door: an answer served yesterday carries a citation opened today, possibly forwarded to someone else, possibly after the document’s tier changed. So requirement 3: opening a citation is an access check against the current manifest, in the opener’s own context — not a static link to a copy.

This is the same rule the previous training applied to served answers leaving a boundary, now applied to sources: the check travels with the artifact, because the artifact outlives the moment it was authorized.

⚠The audit that makes this real

The prompt’s final demand — list every path to the model that bypasses the filter — is the difference between having a control and believing you have one. Real bypasses found this way: a debug endpoint that queries the raw index; the evaluation harness itself (it queries as nobody, seeing everything); the re-index job logging sample passages into a world-readable log. Each is a path where “in the corpus” quietly becomes “visible.” The list must be empty, or the finding routed.

Stop and escalate when the asker context itself is unavailable — the serving surface cannot say who is asking, or roles do not map to tiers. That is an identity-infrastructure gap, owned by the platform team, and the honest interim is serving only the all-staff classes: shipping restricted classes on an honor-system asker context is 2.3’s incident with extra steps.

KNOWLEDGE CHECK

A restricted passage is retrieved for an unauthorized asker but dropped from the results before the model sees it. The ranking, however, was computed over all candidates. What information can still leak?

Key takeaway

The tier from corpus design becomes real at exactly one point: the candidate search, where a restricted passage is not dropped late but never a candidate at all — a model cannot leak what it never received, and that structural guarantee is indifferent to every model failure mode. Filtered queries stay silent about what they filtered, because existence is information; citations re-check access against the current manifest on every open, because artifacts outlive their authorization. Then audit every path to the model, and route what the audit finds. Lesson 3.4 asks the question all of this was built to answer: did retrieval find the evidence at all?

LEADERSHIP DECISION
accept only the structural control - the filter in the candidate search - and treat any instruction-based alternative as no control
PRACTITIONER ACTION
filter candidates, not results; keep filtered queries silent; re-check access on every citation open; audit every path to the model
SUCCESS MEASURE
the bypass-path list is empty and re-audited on change; the same question asked at two tiers provably sees two corpora
Search lessons