Contracts and Control Planes

The last workload standing

Meridian’s apps have destinations. What remains is the heaviest thing in the estate: the data. Ten years of nightly extracts on the reporting share, plus the historical tables in Oracle that feed them — all headed for a data lake (columnar files + a serverless SQL layer over them, rung 2 on the ladder).

The plan uses a pattern worth knowing on its own: the bootstrap loader. Meridian’s managed data-integration service bills per activity and is excellent at incremental syncs — but pointing it at ten years of history would be slow and expensive. So a commodity VM does the one-time historical backfill with plain code, then writes the integration service’s own bookkeeping so that the service resumes as if it had done the load itself. The docs for the original of this pattern call it, cheerfully, “hijacking the control plane.” The handoff mechanics are Lesson 3.2. Today: the configuration architecture that keeps a migration of hundreds of tables honest.

Three planes, kept apart

Migrations rot when three different kinds of truth get tangled in one config file. Keep them separate and name them:

1. The recipe — operator intent. Which objects to move, what filters apply, batch sizes, full-load versus delta mode. Recipes are saved and forkable: when three tables out of two hundred need re-extraction (Lesson 3.3), you fork the master recipe down to those three — same engine, same settings, surgical scope. You never edit the master to do a remediation.

2. The data dictionary — the external contract. Owned outside the migration team, this declares each object’s authoritative field list and filter. It is the answer to “what does correct look like?” — which makes it the reference that reconciliation validates against in Lesson 3.3. The migration reads the contract; it never writes it.

3. The control table — runtime bookkeeping. One row per object in the integration service’s own store: source identity, loading behavior (including the watermark), sink location. This is the plane the bootstrap loader writes into — carefully, transactionally — so the managed service can pick up where the backfill left off.

Design the configuration layer for a bulk data migration (200+ objects,
relational sources to lake storage). Keep three planes strictly separate:
- RECIPE: operator intent (objects, filters, batch size, mode); forkable
- DICTIONARY: externally-owned contract (authoritative fields + filter per object)
- CONTROL TABLE: runtime bookkeeping (source identity, watermark, sink path)
Specify: which plane wins on disagreement, and what the engine does when
recipe and dictionary disagree. Default to failing closed.

Fail closed on drift

That last line of the prompt is the heart of the design. When the recipe says “extract WHERE region = ‘WEST’” and the dictionary says the contract filter is “region IN (‘WEST’,‘CENTRAL’)”, the engine has detected contract drift — and the correct behavior is to abort loudly before extracting a single row, with an explicit override flag for deliberate exceptions.

The alternative is quiet: the run completes, green dashboards, and files in the lake that mean something subtly different from what every consumer believes. Quiet wrongness in a data migration compounds — every downstream report inherits it. A loud stop costs an hour; a quiet drift costs the reconciliation weeks later, when nobody remembers which run was which. Lesson 3.3’s manifests exist for exactly that forensic moment.

Error classification: permanent before transient

A 200-object run will hit errors. What separates a professional engine from a script is one classifier, consulted everywhere, with a strict evaluation order:

PERMANENT (never retry):  auth failures, permission denied, object missing,
                          SQL syntax errors, "your request is wrong" HTTP codes
TRANSIENT (retry):        timeouts, connection resets, throttling, 5xx,
                          DNS hiccups, TLS handshake failures

RULE 1: check PERMANENT indicators BEFORE transient ones.
RULE 2: classify on exception TYPES, not message substrings.
RULE 3: on retry exhaustion, re-raise the ORIGINAL exception.
RULE 4: log every retry before sleeping.

Rule 1 is the subtle one: an error text like “connection timeout during authentication: invalid credentials” contains both a transient marker (timeout) and a permanent one (invalid credentials). Check transient first and you retry a bad password until lockout — now your migration has caused an incident in the source system. Permanent-first means the fatal fact wins.

Two refinements from production scar tissue: some failures earn empirical carve-outs — one real engine classifies truncated-JSON responses from a SaaS API as transient, because the server demonstrably truncates under load and the identical request succeeds on retry; and failures are recorded at chunk granularity, so a 40-million-row object that failed in one chunk shows as “99.2% loaded, chunk 217 failed” — a targeted re-pull — rather than “object failed — restart everything.”

Key takeaway

Three planes, never tangled: recipes carry intent (and fork for remediation), the dictionary carries the contract (and is the reference for “correct”), the control table carries runtime state (and is written transactionally). Disagreement fails closed and loud. Errors classify permanent-before-transient, on types. On this foundation, Lesson 3.2 executes the trickiest move in the pattern: the watermark handoff.