Which assumptions in the estimate need testing?
Updated
BeginnerWhat you'll learn
~11 min- Name the phases of an estimate that a new tool could change and the phases it does not
- Ask for the old estimate split by phase and keep it for comparison with completed comparable work
- Decide not to treat a vendor demo time or a one-week claim as a schedule
You leave this lesson with: a one-page list of the estimate’s phases, each marked “test” or “keep” with the evidence that would settle it.
Keep the old estimate for comparison
A state agency program office has an estimate on file for a case-tracking upgrade: a delivery date and a stated budget. The team wrote it from experience, using what they knew about the work.
That estimate is not a mistake to be corrected. The estimate records what was forecast. Completed comparable work supplies measured performance. A new tool is a reason to test parts of it, not a reason to discard it. If you cannot say what the old estimate assumed, you cannot say what has changed.
Split the estimate before you test it
If the estimate is one number, ask the team to break it into the phases below, and ask which phases a new tool could plausibly shorten.
| Phase | What the time buys | Could AI tools change it? |
|---|---|---|
| Drafting and coding | A first working version of the software | The phase to test first |
| Data preparation | Cleaning, mapping and gaining access to the records | Test; it depends on the data |
| Security and privacy review | A reviewer confirms the use is safe for the data involved | Keep until a test shows otherwise |
| Integration | Connecting to the systems people already use | Test; it depends on the systems |
| Validation | Checking that the output is right against known cases | Keep; it needs a person who knows the answer |
| User adoption | Training, support and a change in daily habits | Keep; people set this pace |
A security reviewer protects the people whose records are involved, and validation protects the people who rely on the output. Faster drafting does not remove what those reviews protect; it changes how much of the schedule drafting occupies.
So a tool that shortens drafting and coding shortens one row. A shorter drafting phase does not establish how much the complete schedule changes. Your team’s own history, such as the last three projects broken out by phase, shows which parts can change and which remain.
Two rules for every claim you hear
No project should be assumed to take a week. A prototype can appear in days, and that is worth knowing. It is not a statement about a finished service that is secured, connected, validated and in daily use.
A vendor demo time is never a schedule. A demo can show a prepared task, on prepared data, with the review, integration and adoption steps left out. Ask three things: what was prepared in advance, what was left out, and whether the same task can run on a sample of your own data against your current handling time. Then make Lesson 2.3’s request: show it working against how the work is done today, and show the written plan before anything is built.
What measured evidence says, and does not say
How fast work feels and how fast it is can differ, so ask for a measurement.
In a randomized trial (METR, July 2025, 16 experienced developers from large open-source repositories), developers believed afterwards that AI had made them about 20% faster. Measured, their tasks had taken 19% longer.
Its follow-up (METR, February 2026, 57 developers) estimated that developers took less time with AI.
METR itself now calls the 19% slowdown out of date, and calls its follow-up very weak evidence for the size of the speedup, so neither result is a figure you can plan on. What still holds is the gap between belief and measurement: for those developers on those tasks, “it feels faster” and “it was faster” differed.
Published studies measure individual tasks under controlled conditions, and results vary widely by task. Do not apply an outside percentage to your schedule. Use a time-boxed demonstration to measure how much of your own workflow changes, and how much time remains in review, integration and approval.
Pick the one phase of the estimate where a tool is most likely to help, set a short time box, and record the calendar time before and after. Agree a time box with the team lead that fits the task and available review capacity, and compare the result with the same phase on the last comparable project. If no record exists for that phase, this run becomes the record. A measured result for one phase supports a claim about that phase. Measure the complete workflow before claiming a change in delivery time.
A vendor shows a working prototype built quickly. What does that tell you about your schedule?
Key takeaway
- LEADERSHIP DECISION
- Keep the existing estimate for comparison and require it to be split by phase before any phase is shortened.
- IMMEDIATE ACTION
- Ask the delivery lead this week which phase a tool might change and how it will be timed against the last comparable project.
- SUCCESS MEASURE
- Calendar days for the tested phase against the same phase on the last comparable project.