Leading in the AI Era Module 3 · Lesson 3.2 · Recognizing organizational barriers

Resolving concerns with evidence: what would settle the question?

Updated

Beginner

What you'll learn

~12 min
  • Record the specific risk, the accountable reviewer, the evidence that would settle it and the decision date for any open concern
  • Ask the reviewer to explain the risk and what evidence would settle it
  • Decide to stop a pilot when a security or privacy concern is right

You leave this lesson with: a four-line record for each open concern: the risk, the accountable reviewer, the evidence that would settle it, and the decision date.

A concern is a question that has not been answered

A reviewer raises a concern about a pilot. It might be about security, privacy, accuracy, cost or fit with policy. The reviewer exists to raise exactly such things, and some concerns will be right.

Ask the reviewer to explain the risk and what evidence would settle it. Understanding the concern helps define the evidence request; speculation about the reviewer’s motive does not.

The four-line record

Record these four things for any open concern.

  1. The specific risk. One sentence naming what could go wrong and for whom. “Customer records could be exposed to the vendor” is a risk. “Security has reservations” is not.
  2. The accountable reviewer. One named person, with the authority to say the risk is acceptable or not. A committee is not a name.
  3. The evidence that would settle it. Document review, an approved test, or both. For a document review: which documents, checked by whom. For a test: which data it uses, how many users, what is measured, what result would count as acceptable. A test on synthetic data, invented records shaped like the real ones, shows how the tool behaves; it cannot settle whether real data may be shared.
  4. The decision date. The calendar day on which the reviewer decides, with what is known by then. Set it before the test starts. If the evidence is incomplete on the day, the reviewer decides on what exists or records an extension with a new date; the date never drifts without a record.

When the reviewer cannot say what evidence would settle the concern, that is the first thing to resolve. Ask the question directly: “What would you need to see?”

An example from a company

A mid-sized manufacturer’s finance team wants to try an AI tool to draft first-pass explanations of monthly variances. The privacy lead has raised a concern about supplier contract terms appearing in the drafts. The four lines read:

  • Risk: supplier contract terms, which are confidential, could be sent to the tool provider’s systems.
  • Accountable reviewer: the privacy lead.
  • Evidence that would settle it: the provider’s written retention and reuse terms, checked by the privacy lead before any data is shared, which settle permission; and a time-limited trial on approved synthetic files, which shows how the drafts behave.
  • Decision date: the Friday after the trial ends.

The concern now has an owner, an evidence request and a date. The privacy lead decides whether the next step is an approved test, a revision, postponement or stopping.

When the concern is right

Not every concern ends in a pilot. Sometimes the evidence shows that the risk is real and cannot be reduced at acceptable cost. Then the pilot stops. Record an evidence-based stop as a valid decision, and make clear that reviewers are expected to raise concerns.

💬Decision: privacy concern on a pilot

A state agency’s caseworker support team proposes a pilot that summarizes case notes containing health information, and the privacy officer has determined that retaining those notes in the vendor’s system is unacceptable for this use. The vendor’s written terms confirm retention, and the team has already found that removing the health information cannot be done at acceptable cost.

  1. Proceed, and ask the privacy officer to document the concern in the project file.
  2. Record the four lines and stop this pilot, because the proposed use cannot meet the privacy requirement at acceptable cost.
  3. Record the four lines, try again to remove the health information from the case notes with no new method or evidence, and decide on the date set.

Pick one before you open What happens.

ℹWhat happensclick to expand
  1. Proceed and document the concern: It keeps the team’s work moving and puts the concern on record, but it does not resolve it; the risk remains, and the agency holds the notes in a system its privacy officer has not approved.
  2. Stop this pilot: The pilot stops, and the agency holds a dated, owned record showing it declined to put health information into a system its privacy officer had not approved.
  3. Try again to remove the health information: It repeats an attempt that has already failed at acceptable cost, with nothing new that could change the result, so it spends staff time and delays the decision without adding evidence.

Choice 2 holds up, because the privacy requirement is explicit and the proposed way to meet it has already failed at acceptable cost. Choice 3 would hold up only if the team had a new way to remove the health information, or new evidence that removing it could now succeed. Line 3 of the record, the evidence that would settle it, is where that would be written; when line 3 holds nothing new, the attempt is a repeat.

KNOWLEDGE CHECK

A reviewer raises a security concern about a pilot. What should the leader ask first?

Key takeaway

LEADERSHIP DECISION
Require the four-line record for every open concern before a pilot continues or stops.
IMMEDIATE ACTION
Ask each reviewer with an open concern what evidence would settle it, and write down the answer with a decision date.
SUCCESS MEASURE
Open concerns reaching a recorded decision of any kind by their decision date.
Search lessons