Leading in the AI Era Module 6 · Lesson 6.2 · What a leader should do next

Measure outcomes, not activity

Updated

Beginner

What you'll learn

~9 min
  • Name the four measures to require: user outcomes, error and rework rate, end-to-end elapsed time, and total cost by category
  • Name the four activity measures to reject: licenses bought, prompts sent, meetings held and code volume
  • Compare single-task time saved with end-to-end time saved for the same workflow

You leave this lesson with: the measurement lines of the worksheet, filled with the four measures the pilot reports at day 60 and day 90 and a source for each baseline.

Four measures worth requiring

User outcomes. What did the people who use the result get? A resident gets a permit decision, a researcher gets a finished report, a customer gets a corrected invoice. Ask the team to state the outcome in the user’s terms and to collect it from users directly, by a short survey or a sample of completed cases.

Quality, as error rate and rework rate. How often is the output wrong, and how often does someone have to redo it? Count both before the pilot and during it. A faster output that is corrected twice as often has moved the work, not removed it.

End-to-end elapsed time. Calendar time from the moment the request arrives to the moment the finished result reaches the user, for the whole workflow. This includes waiting for review, approval and handoff.

Total cost, by category. The same five categories as the worksheet: implementation (including baseline collection and evaluation), review, training, support and ongoing operation. Cost is one signal, not a verdict. A pilot can cost more and still be the right decision if the outcome is worth it, and the reverse.

Four measures never to use

Activity is easy to count and says little about whether the work changed.

  • Licenses bought. It measures a purchase, not a change in the work (Lesson 1.1: licenses alone do not change work).
  • Prompts sent. It measures use. People send many prompts when the tool is helping and when it is failing them.
  • Meetings held. It measures coordination effort, which may mean a program is healthy or stuck.
  • Code volume. It measures output size. More code is more to review, maintain and secure, and it does not say whether the user’s problem was solved.

These numbers are useful as diagnostics: a team that bought licenses nobody uses has learned something. They never appear as the result.

Single-task time versus end-to-end time

A tool can save real time on one step and still leave the workflow no faster. The measure that matters to the user is the whole workflow.

ℹWorked example: a research group's grant progress reports

A university research group sends progress reports to its funder. Its workflow includes collecting results, drafting, investigator review, sponsored-programs sign-off and submission.

The drafting tool may shorten drafting while increasing correction work during review. Sign-off by sponsored programs, the university office that checks the report against the funding agreement, has its own waiting time.

Record drafting time and request-to-submission time separately against the group’s own baseline. Report the whole-workflow result to the dean, including any added review work.

Measure the steps of your own workflow before and during the pilot, and report the sum.

KNOWLEDGE CHECK

A pilot shortens drafting but adds correction work during review. What should the team report?

Where cost goes next

Cost as one signal among several is covered in Operating in Production (Lesson 2.4, a separate training in this series). The cost of a whole portfolio of AI systems and how to read the bill is covered in AI Assurance (Lesson 6.3, a separate training in this series).

Key takeaway

LEADERSHIP DECISION
Require end-to-end elapsed time, rework rate, user outcome and cost by category as the only results reported at day 60 and day 90
IMMEDIATE ACTION
Ask the team for the workflow's elapsed days from request to delivery today, step by step, with the waiting days shown separately
SUCCESS MEASURE
Request-to-delivery calendar days against the baseline at the review, with user outcome no worse than before
Search lessons