Leading in the AI Era Module 1 · Lesson 1.3 · What these systems can and cannot do for your organization

The limits, plainly

Updated

Beginner

What you'll learn

~9 min
  • Name three limits of these systems in terms of what a leader is responsible for
  • Decide when a use should be postponed or cancelled because a limit cannot be bounded (limited to stated data, users and actions) or checked
  • Ask for the evidence that would settle whether a cited answer holds up

You leave this lesson with: a one-line note per proposed use stating which limit applies and how it will be checked.

Limit one: fluent falsehoods, even with sources cited

AI tools that work from plain-language instructions, these systems in the rest of this lesson, write fluent, confident text whether or not it is true. A citation does not close the gap. A cited source can be real and still not say what the answer claims, or can be out of date.

For a leader, that means deciding how answers are checked against their sources (every answer, a sample, or those above a consequence level) and who checks. The cost of that checking is part of the cost of the use. Grounded Answers (a separate training in this series) covers how teams test whether answers are supported.

Limit two: actions across whatever the tool can reach

A system connected to other tools may be able to send messages, change records or make purchases, depending on its permissions. Ask which actions are actually enabled and what damage an incorrect action could cause. Deloitte’s survey of 3,500 senior leaders reports that 40% allow AI to execute decisions or actions without human approval to at least a moderate degree (Deloitte, State of AI in the Enterprise, senior-leader self-report). That figure is a reason to ask the question below before a tool is connected, not after.

The leader’s question is what the tool can reach, and who can undo what it does. Guarded Automation (a separate training in this series) covers how those limits are set.

Limit three: no local context unless supplied

The system does not know your agency’s rules, your customer’s history or why a past decision was made, unless someone provides that material. Without it, it answers from general patterns, which may be wrong for you.

Supplying context takes work, and the context itself may be sensitive. Who may share what with the tool is a data decision. AI Assurance (a separate training in this series) covers how an organization decides what evidence a use needs before release.

When to postpone or cancel

When a limit cannot be bounded or verified at acceptable cost, postponing or cancelling the use is the right call. Postponing also has a cost. Record that cost alongside the risk of proceeding, and set a date to revisit the decision if the evidence changes.

💬Decision: a demo that cites sources

A government agency’s records office is shown a tool that answers retention questions and cites a document for every answer. The documents are scanned images of varying age, and the office has limited staff capacity to check them.

  1. Approve a pilot for all staff, because every answer has a source.
  2. Cancel this proposal for the current budget cycle.
  3. Postpone the use until the office can show how citations will be checked at its staffing level.

Pick one before you open What happens.

ℹWhat happensclick to expand
  1. Approve a pilot for all staff: It puts the tool in front of staff quickly, and every answer does carry a source. But staff rely on answers whose citations point to scans nobody has time to read, and a wrong retention answer is found later, in an audit.
  2. Cancel this proposal: It keeps wrong answers away from staff and frees the budget, but it also gives up a use that could work once checking is affordable, and the office never learns what checking would take.
  3. Postpone the use: The office writes down what checking one citation takes and finds it is more staff time than it has. It postpones with that measured cost in hand and a date to revisit.

Choice 3 holds up, because the sources cannot be checked at acceptable cost with the current staff and scans. The office pays the cost of waiting and keeps the option to return when the checking cost changes.

Optional deeper reading: Grounded Answers leadership brief, one page

Key takeaway

LEADERSHIP DECISION
Require that each proposed use names which of the three limits applies and how it will be checked.
IMMEDIATE ACTION
Ask the team to write down what it takes to verify one answer from the tool, and who would do it.
SUCCESS MEASURE
Staff-hours to verify one answer, against the checking time the team has available.
Search lessons