Evaluate and decide: expand, revise, defer or stop
Updated
BeginnerWhat you'll learn
~13 min- Name the operational owner and the stop condition before the pilot starts
- Name the four conditions for routine use: user acceptance, staff training, support ownership and operating budget
- Decide at the review among expand, revise, defer and stop, and record the reason
You leave this lesson with: a five-line pilot charter, signed before day one, that names the owner, the bar, the stop condition, the review date and the conditions for routine use.
Write the charter before day one
A pilot needs its ending written down as well as its start date. A charter fixes the following decisions before work starts.
- The operational owner. One named person who will run the capability if it works, not only the person who builds the pilot. If no one will own it, the pilot is a demonstration.
- The bar. What result, measured against the team’s own baseline (Lesson 2.1: the current workflow, measured before the change), counts as success. State it as elapsed time, error or rework rate, or a measured user outcome, and compare measured results, not impressions, because a pilot can feel faster to its users than it measures (Lesson 2.2 describes a trial where it did).
- The stop condition. The result, the date, or the staff-time and spending limit at which the pilot ends.
- The review date. The day a named decision owner chooses among expand, revise, defer or stop.
- The conditions for routine use. Listed next.
If the pilot is already running without a charter, write the five lines now, take the baseline from records dated before the pilot, and set the review date.
Evaluation method belongs to the technical teams. Grounded Answers (Lessons 1.2 and 1.4, a separate training in this series) shows how an engineering team designs the evaluation and sets the kill gate, the point at which a build is stopped. The leader’s part is to make sure the bar and the stop condition exist, in writing, before the pilot begins.
Four conditions for routine use
Routine use requires all four conditions.
- User acceptance. The people who do the work have used it on real work and say it helps.
- Staff training. Users have been shown how to use it on their own tasks, with time set aside to learn (Lesson 4.3 covers protected learning time).
- Support ownership. Someone answers when it breaks or gives a wrong result, on a stated schedule. Operating in Production (a separate training in this series) covers handing a pilot to a live service, including on-call and incidents.
- Operating budget. The recurring cost of running, supporting and reviewing the capability has a line in a budget, not only the cost of the pilot.
Four outcomes, all legitimate
Expand when the bar is met and the four conditions hold. Revise when the result is close and one change would plausibly close the gap; a revision names that one change, its owner, a resource cap and a new review date, and moving only the date is not a revision. Defer when the bar is met but a condition, such as budget or support, is not yet in place; the decision names who resolves the gap and by when. Stop when the bar is missed and revising would not change that.
When the evidence cannot settle whether the bar was met, the pilot does not expand. The owner records why the bar could not be judged, names the missing evidence, who collects it and a cap on the time or money spent collecting it, and sets one more review date. If that review still cannot judge the bar, the decision is stop.
Stopping at the review is a successful outcome. The organization spent no more than the charter allowed to learn that this approach does not pay at this problem, and it did so on the date it said it would. The cost that damages an organization is the pilot nobody stops.
A state agency program ran a time-limited pilot of an AI tool that drafts first responses to public inquiries. The charter set the bar as a shorter time to a correct response, and the review found that staff time spent correcting drafts matched the time saved drafting.
- Stop the pilot as the charter says, record what was learned, and return to the workflow problem.
- Extend the pilot another quarter to give the tool more time to improve.
- Expand to a second office, on the view that scale will make the savings visible.
Pick one before you open What happens.
ℹWhat happensclick to expand
- Stop the pilot as the charter says: The program keeps its credibility and its budget for a better-chosen problem.
- Extend the pilot another quarter: It gives the tool another chance, but it moves the stop date with no new reason to expect a different result, and the charter stops meaning anything.
- Expand to a second office: It repeats a measured miss in more places, and adds support and training costs to a result that has not paid for itself.
Choice 1 holds up, because the bar was written in advance and it was missed.
A pilot meets its bar. No one has been named to support it after the pilot team moves on. What is the sound decision?
Key takeaway
- LEADERSHIP DECISION
- Do not start a pilot until the charter names the operational owner, the bar and the stop condition
- IMMEDIATE ACTION
- Ask the team for the five-line charter and the date of the review, and put the review on your calendar
- SUCCESS MEASURE
- Calendar days from pilot start to a recorded expand, revise, defer or stop decision, against the review date in the charter