Deploy on the High Side
Updated
AdvancedWhat you'll learn
~15 min- Design a parameterized high-side deployer: discover, validate by type, then apply
- Handle IaC state that travels in the package - and the discipline that makes that safe
- Close the no-rollback gap with pre-apply capture and a tested reverse path
The last mile, unattended
The package crossed (5.1), the transfer verified it (5.2); now something has to run it - on agents inside the enclave, operated by people who didn’t build it, against infrastructure you can’t see. The reference implementation is a parameterized deployment pipeline for exactly this seat, and its pipeline structure is a study in designing for the operator you’ll never meet:
THE HIGH-SIDE DEPLOYER: 1. skip source checkout there is no source here - the package is the truth 2. fetch NEWEST package from the landing container, by convention: latest-by-name 3. extract + AUTO-DISCOVER find the IaC in the archive: which tool (template-vs-config priority rules), which parameter files 4. TYPE-GATED VALIDATION per discovered tool, the dry-run that tool offers (what-if / plan) - results surfaced BEFORE apply 5. apply then summarize; on failure - log (and there is the gap)Two design choices deserve their moment. Auto-discovery makes the deployer generic: it finds .bicep or .tf, detects parameter files, chooses validation per type — so one pipeline serves every package that follows the convention, and the convention (not documentation) is the interface. Type-gated validation honors an invariant this training has hammered: never apply what you haven’t dry-run in the target environment — the low-side rehearsal (5.1) proved the package internally coherent; only the high-side plan proves it coherent here.
State travels in the package
The quiet radical decision: IaC state rides inside the archive. No state backend spans the gap (that’s the point of a gap), so the first package seeds the state file, and from then on state lives on the high side, updated in place by each apply — subsequent archives carry configuration, never a replacement state file (the low side can never know post-apply state — resource IDs and computed attributes don’t come back across a one-way gap — so a package that tried to carry authoritative state would be stale by the second cycle, and its apply would try to drag the enclave backward). Protect that high-side state like the secret-bearing artifact it is: locked, backed up before every apply, never exported. This works — the reference system runs it — under one discipline that is really Lesson 5.2 wearing infrastructure clothes: the state is only truthful if every change flows through the one-way pipeline. Any out-of-band high-side change makes the packaged state a liar, and IaC tools act on what state tells them — the next apply will try to “correct” reality back toward stale state, which in an enclave can mean deleting something an operator built for a reason. Drift detection (a plan run that should show no changes when nothing was transferred) is therefore not hygiene here; it’s the alarm that the one-way property broke somewhere upstream.
The gap: failure only logs
The reference deployer’s honest wart, called out by its own operators: on failure, it logs - there is no rollback. A half-applied infrastructure change in an enclave, with the build team on the other side of a guarded gate and the next transfer window days away, is the worst room in this training. Closing the gap is your exercise, and the structure of the close looks like this:
BEFORE apply: capture current state + a reverse-plan artifact (what would undo this change) while the information to compute it still existsDURING: apply in dependency-ordered stages with a checkpoint after each - failure names the completed stagesAFTER failure: the operator gets a DECISION, not a log: retry the failed stage / execute the captured reverse path / hold - each as a runnable command, with REVERSIBILITY marked per stage (deletions, data migrations, and credential rotations may have no reverse path - say so BEFORE the stage runs, not after), in the runbook the package carried (5.1's rule: the runbook rides inside the tooling)The deeper point generalizes past enclaves: rollback is a plan you capture before you need it, not a feature you discover you lack. The moment to compute “how do I undo this” is while the pre-change state still exists — after failure, half the information is gone. (The deploy-decoupled-from-release discipline — flags, progressive rollout, five-minute reversibility — is the DevSecOps Foundations training’s lane; here the constraint is harder: no traffic-shifting between sides of an air gap. The reverse path must be in the package.)
Prompt first: close it yourself
Switch your workspace agent to Plan mode on your high-side deployer YAML and scripts, and ask it to plan the failure-path hardening (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):
Review our high-side deployer pipeline YAML and scripts. Plan how to add pre-apply state and reverse-plan capture, staged checkpoints, an out-of-band drift alarm, and a low-side rollback rehearsal.Check that the plan or draft covers:
- Capturing a current state snapshot and a generated reverse plan beside the package’s
evidence/folder before any apply touches the enclave - Dependency-ordered apply stages with named checkpoints that emit runnable operator commands (
retry-stage,reverse, orhold) with preconditions and irreversibility warnings stated up front - A scheduled drift-detection plan run that alerts when changes appear with no transfer since the last run
- A low-side rehearsal that deliberately breaks a staged apply and proves the captured reverse plan restores the prior state
If the rollback steps are written as prose advice instead of runnable commands, follow up with: Emit the post-failure retry, reverse, and hold options as exact runnable CLI commands with their preconditions.
That fourth requirement is the one that separates a rollback plan from a rollback hope: a reverse path that has never executed is documentation, not capability.
The deployer finds packages by a naming convention in a landing container - which means the transfer pipeline (5.2) and the deployer are coupled by CONVENTION, not by contract. The reference programs treat this seam honestly: the convention is documented once, tested from both sides (the transfer emits a conformance check; the deployer validates on pickup), and versioned when it must change. Unwritten conventions between separately-owned systems are where unattended pipelines go to die quietly - if you own one side of such a seam, write the convention down and test it from your side even if the other side will not.
A scheduled drift check on the high side reports changes pending - a firewall rule the plan wants to remove - but no package has been transferred in two weeks. What happened, and what is the correct response?
Key takeaway
The high-side deployer is convention-driven and unattended: newest package, auto-discovered IaC, type-gated dry-run, then apply — with state traveling in the archive, truthful only while every change rides the one-way flow (drift with no transfer is your alarm, not noise). Close the reference design’s honest gap before it closes on you: capture the reverse path pre-apply, stage with checkpoints, hand operators decisions instead of logs, and rehearse the rollback until it is capability rather than hope. Module 5 complete — the software is across, running, and recoverable. Next: the loop back toward continuous.