Leading in the AI Era Module 2 · Lesson 2.2 · Rethinking time, cost and capability

Which assumptions in the estimate need testing?

Updated

Beginner

What you'll learn

~11 min
  • Name the phases of an estimate that a new tool could change and the phases it does not
  • Ask for the old estimate split by phase and keep it for comparison with completed comparable work
  • Decide not to treat a vendor demo time or a one-week claim as a schedule

You leave this lesson with: a one-page list of the estimate’s phases, each marked “test” or “keep” with the evidence that would settle it.

Keep the old estimate for comparison

A state agency program office has an estimate on file for a case-tracking upgrade: a delivery date and a stated budget. The team wrote it from experience, using what they knew about the work.

That estimate is not a mistake to be corrected. The estimate records what was forecast. Completed comparable work supplies measured performance. A new tool is a reason to test parts of it, not a reason to discard it. If you cannot say what the old estimate assumed, you cannot say what has changed.

Split the estimate before you test it

If the estimate is one number, ask the team to break it into the phases below, and ask which phases a new tool could plausibly shorten.

PhaseWhat the time buysCould AI tools change it?
Drafting and codingA first working version of the softwareThe phase to test first
Data preparationCleaning, mapping and gaining access to the recordsTest; it depends on the data
Security and privacy reviewA reviewer confirms the use is safe for the data involvedKeep until a test shows otherwise
IntegrationConnecting to the systems people already useTest; it depends on the systems
ValidationChecking that the output is right against known casesKeep; it needs a person who knows the answer
User adoptionTraining, support and a change in daily habitsKeep; people set this pace

A security reviewer protects the people whose records are involved, and validation protects the people who rely on the output. Faster drafting does not remove what those reviews protect; it changes how much of the schedule drafting occupies.

So a tool that shortens drafting and coding shortens one row. A shorter drafting phase does not establish how much the complete schedule changes. Your team’s own history, such as the last three projects broken out by phase, shows which parts can change and which remain.

Two rules for every claim you hear

No project should be assumed to take a week. A prototype can appear in days, and that is worth knowing. It is not a statement about a finished service that is secured, connected, validated and in daily use.

A vendor demo time is never a schedule. A demo can show a prepared task, on prepared data, with the review, integration and adoption steps left out. Ask three things: what was prepared in advance, what was left out, and whether the same task can run on a sample of your own data against your current handling time. Then make Lesson 2.3’s request: show it working against how the work is done today, and show the written plan before anything is built.

What measured evidence says, and does not say

How fast work feels and how fast it is can differ, so ask for a measurement.

In a randomized trial (METR, July 2025, 16 experienced developers from large open-source repositories), developers believed afterwards that AI had made them about 20% faster. Measured, their tasks had taken 19% longer.

Its follow-up (METR, February 2026, 57 developers) estimated that developers took less time with AI.

METR itself now calls the 19% slowdown out of date, and calls its follow-up very weak evidence for the size of the speedup, so neither result is a figure you can plan on. What still holds is the gap between belief and measurement: for those developers on those tasks, “it feels faster” and “it was faster” differed.

Published studies measure individual tasks under controlled conditions, and results vary widely by task. Do not apply an outside percentage to your schedule. Use a time-boxed demonstration to measure how much of your own workflow changes, and how much time remains in review, integration and approval.

💡Test one phase at a time

Pick the one phase of the estimate where a tool is most likely to help, set a short time box, and record the calendar time before and after. Agree a time box with the team lead that fits the task and available review capacity, and compare the result with the same phase on the last comparable project. If no record exists for that phase, this run becomes the record. A measured result for one phase supports a claim about that phase. Measure the complete workflow before claiming a change in delivery time.

KNOWLEDGE CHECK

A vendor shows a working prototype built quickly. What does that tell you about your schedule?

Key takeaway

LEADERSHIP DECISION
Keep the existing estimate for comparison and require it to be split by phase before any phase is shortened.
IMMEDIATE ACTION
Ask the delivery lead this week which phase a tool might change and how it will be timed against the last comparable project.
SUCCESS MEASURE
Calendar days for the tested phase against the same phase on the last comparable project.
Search lessons