Contracts and Control Planes
Updated
AdvancedWhat you'll learn
~20 min- Separate a migration's three configuration planes: operator intent, external contract, and runtime bookkeeping
- Design fail-closed behavior for contract drift
- Classify migration errors as permanent or transient -- in the right order
The last workload standing
Meridian’s apps have destinations. What remains is the heaviest thing in the estate: the data. Ten years of nightly extracts on the reporting share, plus the historical tables in Oracle that feed them — all headed for a data lake (columnar files + a serverless SQL layer over them, rung 2 on the ladder).
The plan uses a pattern worth knowing on its own: the bootstrap loader. Meridian’s managed data-integration service bills per activity and is excellent at incremental syncs — but pointing it at ten years of history would be slow and expensive. So a commodity VM does the one-time historical backfill with plain code, then writes the integration service’s own bookkeeping so that the service resumes as if it had done the load itself. The docs for the original of this pattern call it, cheerfully, “hijacking the control plane.” The handoff mechanics are Lesson 3.2. Today: the configuration architecture that keeps a migration of hundreds of tables honest.
Three planes, kept apart
Migrations rot when three different kinds of truth get tangled in one config file. Keep them separate and name them:
1. The recipe — operator intent. Which objects to move, what filters apply, batch sizes, full-load versus delta mode. Recipes are saved and forkable: when three tables out of two hundred need re-extraction (Lesson 3.3), you fork the master recipe down to those three — same engine, same settings, surgical scope. You never edit the master to do a remediation.
2. The data dictionary — the external contract. Owned outside the migration team, this declares each object’s authoritative field list and filter. It is the answer to “what does correct look like?” — which makes it the reference that reconciliation validates against in Lesson 3.3. The migration reads the contract; it never writes it.
3. The control table — runtime bookkeeping. One row per object in the integration service’s own store: source identity, loading behavior (including the watermark), sink location. This is the plane the bootstrap loader writes into — carefully, transactionally — so the managed service can pick up where the backfill left off.
Switch your workspace agent to Plan mode on your migration repository and ask it to design the three-plane configuration layer:
Plan the three-plane configuration layer (recipe, data dictionary, and control table) for our 200-object bulk data migration from relational sources to lake storage. Default to failing closed when the recipe and dictionary disagree.Check that the plan or draft covers:
- Strict separation of the three planes: forkable operator recipes (objects, filters, batch size, mode), the externally owned data dictionary (authoritative fields and filter per object), and the runtime control table (source identity, watermark, sink path)
- Fail-closed behavior that aborts extraction before pulling a single row whenever a recipe filter or field list drifts from the dictionary contract
- An explicit override flag required for deliberate contract exceptions
If the draft logs a warning and continues when a recipe filter differs from the dictionary, follow up with: Change contract-drift handling to abort loudly before extraction unless an explicit override flag is passed.
Fail closed on drift
That last line of the prompt is the heart of the design. When the recipe says “extract WHERE region = ‘WEST’” and the dictionary says the contract filter is “region IN (‘WEST’,‘CENTRAL’)”, the engine has detected contract drift — and the correct behavior is to abort loudly before extracting a single row, with an explicit override flag for deliberate exceptions.
The alternative is quiet: the run completes, green dashboards, and files in the lake that mean something subtly different from what every consumer believes. Quiet wrongness in a data migration compounds — every downstream report inherits it. A loud stop costs an hour; a quiet drift costs the reconciliation weeks later, when nobody remembers which run was which. Lesson 3.3’s manifests exist for exactly that forensic moment.
Error classification: permanent before transient
A 200-object run will hit errors. What separates a professional engine from a script is one classifier structure, consulted everywhere, with a strict evaluation order:
PERMANENT (never retry): auth failures, permission denied, object missing, SQL syntax errors, "your request is wrong" HTTP codesTRANSIENT (retry): timeouts, connection resets, throttling, 5xx, DNS hiccups, TLS handshake failures
RULE 1: check PERMANENT indicators BEFORE transient ones.RULE 2: classify on exception TYPES, not message substrings.RULE 3: on retry exhaustion, re-raise the ORIGINAL exception.RULE 4: log every retry before sleeping.Rule 1 is the subtle one: an error text like “connection timeout during authentication: invalid credentials” contains both a transient marker (timeout) and a permanent one (invalid credentials). Check transient first and you retry a bad password until lockout — now your migration has caused an incident in the source system. Permanent-first means the fatal fact wins.
Two refinements from production scar tissue: some failures earn empirical carve-outs — one real engine classifies truncated-JSON responses from a SaaS API as transient, because the server demonstrably truncates under load and the identical request succeeds on retry; and failures are recorded at chunk granularity, so a 40-million-row object that failed in one chunk shows as “99.2% loaded, chunk 217 failed” — a targeted re-pull — rather than “object failed — restart everything.”
🔍Why types, not substringsclick to expand
Message-substring matching (“does it contain ‘timeout’?”) is the tempting shortcut, and it breaks in both directions: vendor drivers localize and rewrite messages between versions, and rich error text often contains both classes of keyword at once. Exception types and structured error codes are the API contract; messages are decoration. The carve-outs above are deliberate, documented exceptions to this rule — earned by observation, not convenience. (And note that Rule 1’s example error is a composite — a cause chain carrying both classes at once — which is precisely the case where substring matching betrays you and type-first, permanent-first classification holds.)
Mid-migration, the dictionary owner adds a column to an object's contract. The next run's recipe no longer matches. The engine aborts the object with a contract-drift error instead of proceeding. Why is this the right behavior?
Key takeaway
Three planes, never tangled: recipes carry intent (and fork for remediation), the dictionary carries the contract (and is the reference for “correct”), the control table carries runtime state (and is written transactionally). Disagreement fails closed and loud. Errors classify permanent-before-transient, on types. On this foundation, Lesson 3.2 executes the trickiest move in the pattern: the watermark handoff.