AI Engineering: After the Demo Works

The route

6 stages in dependency order: the question before the build, the proof before the claim, the register before the evaluation, the pager before the agent. Every stop is a whole module — you cannot take half of one here, because the lessons inside each assume the ones before them. Each stage names the test for skipping it: skip only if every line is true of you.

If your roadmap budgeted one "supporting skills" stage, this is what that stage decomposes into. Where a stage's training lists an earlier stage's training as what it builds on, the catalog's own field is printed under the stage, so the order is something you can check rather than take on trust.

1. Stage 01

   The exits before the build

   1 module · 4 lessons · 71 min

   A roadmap's retrieval stage starts with a build. This one starts with four named ways to answer the question that need no build at all, and what it costs to take each. If the question survives that table, you write the evaluation questions — and name who knows the right answers — before anything retrieves.

   Skip this stage if all of these are true

   • you have written down the questions your system must answer and the ones it must refuse
   • you have named who scores them and what the pass bar is
   • you wrote all of that before the first retrieval ran
   • Decide the Answering Method Module 1 of 6 in Grounded Answers From Documents · all 4 lessons · 71 min

     A bounded class of questions, not a chatbot

     Why here First because everything after it is judged against the question set this module makes you write — and because its exits can end the route here, honestly, before any build.

     • The Questions Nobody Can Answer 15m
     • The Kill Gate 18m
     • Bound the Question Classes 18m
     • Freeze the Question Set 20m
2. Stage 02

   Retrieval as a contract, then grounding as a claim

   2 modules · 8 lessons · 152 min

   A question goes in; authorized passages with openable provenance come back — that contract outlives whichever index you pick. Then the part the demos skip: a citation proves where text came from, never that it supports the claim.

   Skip this stage if all of these are true

   • your retriever enforces permissions where the evidence is fetched
   • retrieval is evaluated on its own before any answer exists
   • you sample-check that each cited passage actually contains the claim
   • Retrieve the Evidence Module 3 of 6 in Grounded Answers From Documents · all 4 lessons · 76 min

     Find it, or say so

     Why here Retrieval comes before proof because the hand check in the next module needs passages with provenance to check against — and permissions have to be enforced here, or the proof is of the wrong corpus.

     • The Retrieval Contract 18m
     • The Build 22m
     • Retrieval Respects Permissions 18m
     • Did It Find the Evidence? 18m
   • Prove the Answer Module 4 of 6 in Grounded Answers From Documents · all 4 lessons · 76 min

     Support is not truth

     Why here Placed before any evaluation of the whole system: until you can tell a supported answer from a merely cited one, an evaluation score is measuring the wrong thing.

     • The Hand Check 18m
     • Abstention Is a Feature 18m
     • Which Half Failed 20m
     • The Honest Limits 20m
3. Stage 03

   Name the system before you judge it

   1 module · 4 lessons · 75 min

   Evaluation cannot authorize a system nobody has named. The register, the consequence tier — how much harm a wrong output can do, and so how hard the system must be tested — that a second reviewer must be able to reproduce, and what a release decision actually is. The tier is also what sizes every test in the next stage.

   In the catalog, AI Assurance: System Risk and Release Decisions lists Grounded Answers From Documents under "builds on".

   Skip this stage if all of these are true

   • every AI system you run has a named owner and a version boundary
   • each one has a consequence tier a second reviewer could reproduce
   • a named signer was assigned before testing began
   • List What You Run Module 1 of 6 in AI Assurance: System Risk and Release Decisions · all 4 lessons · 75 min

     You cannot assure what you cannot list

     Why here Before evaluation because the tier this module assigns is what sizes every test in the next stage — and because a system nobody has registered cannot be authorized by any score.

     • Ten Systems, One Signature 15m
     • The AI System Register 20m
     • Consequence Tier and Gate Path 22m
     • The Release Decision 18m
4. Stage 04

   Evaluate on your data, behind a gate

   1 module · 4 lessons · 81 min

   The supporting skill every roadmap lists and few walk. Evaluation is chosen by system type — a classifier needs a threshold and slices, a generator needs planted facts and refusals, an agent needs trajectories — run on a set the provider has never seen, by a gate that exits non-zero.

   In the catalog, AI Assurance: System Risk and Release Decisions lists Grounded Answers From Documents under "builds on".

   Skip this stage if all of these are true

   • your evaluation runs as a gate that can fail a release
   • it runs on data your provider has never seen
   • the method was chosen by system type, not by whatever the demo happened to produce
   • Evaluate on Your Terms Module 3 of 6 in AI Assurance: System Risk and Release Decisions · all 4 lessons · 81 min

     Your data, your gate

     Why here Here because it consumes both earlier stages — the frozen question set from the first and the tier from the third — and produces the gate the pager stage assumes already exists.

     • Evaluation by System Type 22m
     • The Assurance Workbench 25m
     • Compare Candidates on One Contract 16m
     • When You Cannot Test the Model 18m
5. Stage 05

   The pager and the record

   2 modules · 8 lessons · 131 min

   The gap a roadmap's supporting-skills bullet hides: deployment, monitoring, on-call. Not tooling — the human system around a running service. Declare at a low threshold, decide honestly whether you can staff the response, keep one record several different readers can use, and page from a burn rate — how fast the service is spending the errors its service level allows — that has a unit test rather than from a chart.

   In the catalog, Operating in Production: On-Call, Incident Command, and Reporting Clocks lists AI Assurance: System Risk and Release Decisions under "builds on".

   Skip this stage if all of these are true

   • your service has a declared incident threshold
   • it has a rota (the on-call schedule) someone actually signed up for
   • there is one record per incident that every reader writes into
   • alerts derive from a service level, not from a CPU graph
   • The System Around the Service Module 1 of 6 in Operating in Production: On-Call, Incident Command, and Reporting Clocks · all 4 lessons · 62 min

     Built before the first page

     Why here Before the page because declaring, staffing, and the one record are what a page is a claim on; without them a burn-rate alert wakes someone with nowhere to write.

     • Saturday 02:14 12m
     • Declare It 16m
     • Coverage Is a Decision 18m
     • The Incident Record 16m
   • The Page Module 2 of 6 in Operating in Production: On-Call, Incident Command, and Reporting Clocks · all 4 lessons · 69 min

     A claim on someone's night

     Why here Last of the human-system modules because its burn-rate policy needs the service level the previous module made you state — and because this telemetry is what the agent stage requires to exist first.

     • Every Page Is a Claim 15m
     • Burn Rate Is the Page 22m
     • The Rota Is a Contract 18m
     • Cost Is a Signal 14m
6. Stage 06

   Only now, let it act

   1 module · 4 lessons · 60 min

   The roadmap shape this route answers — agents as one stage, the supporting skills as the stage after — is reversed here, and the catalog says why in a field printed under this stage: Guarded Automation lists Operating in Production as what it builds on, because an agent cannot safely run an operation whose telemetry and recovery were never established by people first. What this stage adds is the distinction the field keeps blurring: the envelope — the limits an action is pre-approved to run inside — can stand, but the authorization cannot.

   In the catalog, Guarded Automation: Agents That Run Operations lists Operating in Production: On-Call, Incident Command, and Reporting Clocks and AI Assurance: System Risk and Release Decisions under "builds on".

   Skip this stage if all of these are true

   • every action your agent can take is classified by impact and reversibility
   • that classification is signed by a named owner
   • every action leaves a receipt in a store the agent cannot write to
   • One Action, One Moment Module 1 of 6 in Guarded Automation: Agents That Run Operations · all 4 lessons · 60 min

     Classify before the first run

     Why here Last on purpose: its action register assumes the register, the gate, and the pager above already exist. The catalog encodes that order in the builds-on field printed under this stage.

     • Fourteen Crews and One Instruction 12m
     • The Envelope Can Stand; the Authorization Cannot 15m
     • The Action-Class Decision 18m
     • The Action Receipt 15m