Promotion, Rollback, and Day 2

The last mile

Everything works in dev: FieldDesk’s rebuild, AssetTrack’s containerized API, the lake feeding real reports. Promotion to production is where modernizations stumble at the finish line — not because production is mysterious, but because promotion gets improvised. The fix is the same discipline as everything else in this training: a runbook with gates, executed the same way every time.

The promotion runbook

PRECONDITIONS (all verified, not assumed):
  □ Dev/staging stable: error rate at baseline for 7 days
  □ All migrations already applied and verified in staging
  □ Smoke suite green in staging
  □ Telemetry healthy - dashboards live, alerts wired

PROMOTE:
  1. Config freeze - export staging settings; diff against prod expectations
  2. Migration DRY-RUN against a prod-copy database; review the plan
  3. Mirror settings + vault references to prod
  4. VERIFY the app's workload identity resolves every secret reference
     in prod - before any traffic
  5. Deploy (new revision, prod traffic still on the old one)
  6. Run migrations
  7. Smoke suite against the new revision directly
  8. Shift traffic; watch error rate + p95 against baseline
BACKOUT (prepared before step 5, not invented during an incident):
  - Previous revision kept warm (Lesson 2.3's rollback button)
  - Exported previous settings retained
  - Schema: see below - this is the asymmetry

Step 4 is the runbook’s unsung hero. Lesson 4.1 made configuration reference-only; the classic promotion failure is an app whose managed identity has no vault access in the new environment — everything deploys green and the first request dies resolving a secret. Verifying reference resolution before traffic costs one command and removes the most common first-hour incident.

The asymmetry nobody writes down

Application code rolls back trivially: route traffic to the previous revision, done. Database schema does not. Migrations are forward-only in practice — a “down” script that drops the column you just backfilled destroys data, and restoring last night’s backup discards today’s transactions. So say the asymmetry out loud, in the runbook:

Code rollback:   traffic shift to previous revision (minutes, safe, rehearsed)
Schema rollback: does not exist. Recovery is a NEW forward migration
                 (planned repair), written calmly, tested in staging, promoted
                 through the same gates.

(One scope note: forward-only with expand/contract is the right default for continuously-deployed apps like FieldDesk. Estates with release-train cadences, DBA-owned schemas, or maintenance windows sometimes run a different, equally deliberate school — paired rollback scripts per release. The DevSecOps Foundations training’s database lesson gives the four criteria that pick between them.)

Naming it changes behavior upstream: migrations get written to be compatible one version back (add the column nullable now, enforce it next release — the expand/contract pattern), because the previous revision must be able to run against the new schema for the code-rollback button to be safe at all. And the migrations themselves follow the rules FieldDesk’s rebuild set: versioned, idempotent seeds, valid against both empty and populated databases.

Day 2: the ops you kept

The ladder promised most operations would transfer to the platform. It also promised honesty about the remainder. Meridian’s standing calendar, by rung:

Two rows deserve their asterisks. Restore drills: backup you have never restored is a hypothesis, not a capability — the quarterly drill is what converts it. Cluster upgrades: this row exists only because of the one workload that descended the ladder; it is the recurring price of Lesson 2.4’s exception, exactly as invoiced.

The estate, after

Close the loop with where Lesson 1.1 started:

“Get us out of the VM business without breaking the utility” — done, with one written-down exception and a calendar that keeps it honest.

Key takeaway

Promote through verified preconditions — with secret-reference resolution checked before traffic — keep the backout rehearsed, and write migrations expand/contract because schema only rolls forward. Then run Day 2 off a calendar sized to each rung, restore drills included. That closes the first arc: assess, decide, place, containerize, migrate, harden, prove, ship — and operate less than you did before. Two modules remain, and they are where 2026 practice actually lives: using AI to do the modernization at scale (Module 5), and the disciplines that keep a modernized estate from quietly rotting back (Module 6).