Reconcile, Then Cut Over

The gate itself

Two lessons have leaned on one promise: cutover is gated on reconciliation. Today, the gate. Two hundred objects sit in the lake; the incremental service is ready; management wants the file share dead by quarter-end. Reconciliation is how you say “go” with evidence instead of hope.

Single-sided validation

The instinctive design is a diff: compare source row-by-row against the lake. Resist it. Diffing two moving systems yields differences you then spend days explaining — was that mismatch drift since the extract, a timing artifact, or a real loss? The pattern that scales is single-sided validation: each plane is checked independently against the contract (Lesson 3.1’s data dictionary):

Source        vs contract:  row count under the contract's filter; fields readable
Lake files    vs contract:  row counts from file footers; schema matches field list
SQL layer     vs contract:  object queryable; counts consistent with the files

Each check has one authoritative reference — the contract — so each failure names its own plane. Two implementation details make this affordable and honest at 200-object scale:

• Read counts from columnar file footers, not by scanning data. Parquet-style formats store row counts and schema in metadata; validating hundreds of objects via footers takes minutes, not days. Metadata-path reads are what make re-running reconciliation after every remediation cheap enough that you actually do it.
• Count filter-aware, structurally. The classic false alarm: an extract filtered per the contract, compared against an unfiltered source count — panic, then wasted hours. The tooling applies the contract’s filter to the source count every time, so the comparison cannot be run wrong. And when a source uses table inheritance, parent-level counts will “fail” forever: validate at leaf level with children summed, and confirm both sides define the set the same way before comparing at all.

The damage map

Reality by cutover week: some objects were pulled two or three times over the migration’s months — a partial April run, a full May run, a filter-fixed June run for a few stragglers. The naive question, “is the latest pull correct?”, wastes everything earlier. The reframe that saves weeks: “does ANY existing pull satisfy the contract?”

Build the cross-product — object × pull generation, a validation verdict per cell — and most objects turn out to need nothing: some generation already passes. The genuinely damaged residue triages cheapest-first:

Two disciplines inside the table. Class 2’s local transform whitelists only exactly-reproducible filter operators — equality, IN-lists, simple ranges; anything fuzzier raises “unsafe filter” and escalates to class 5, because refusing to approximate is what keeps “re-filtered” meaning “equivalent to a clean extract”. Class 4 is a diagnosis, not a retry: a column the contract promises but the source will not yield is either a phantom dictionary entry or a permissions gap — re-pulling cannot fix either, and hammering the source just delays the human conversation that will.

Manifests: the audit trail

The damage map is only buildable because every run wrote a manifest next to its data:

{
  "object": "WORK_ORDERS", "mode": "bulk", "recipeVersion": "2026-05.2",
  "startedUtc": "2026-05-14T02:00:11Z",
  "watermark": { "from": null, "to": "2026-05-14T02:00:00Z" },
  "filter": "STATUS NOT IN ('PURGED')",
  "fields": ["WO_ID", "ASSET_ID", "OPENED", "CLOSED", "STATUS", "CREW"],
  "counts": { "expected": 41277380, "written": 41277380, "failedChunks": 0 },
  "bytes": 6120843577, "chunks": { "total": 826, "failed": [] }
}

Exact filter, exact field list, exact watermark, expected-versus-written. Reconciliation reads manifests instead of guessing provenance — the difference between “generation 2 used the narrow filter, it’s a class-2 superset case… wait, no, generation 3 is” settled by a file read instead of an archaeology session.

One design tension, preserved because it teaches: the lake had a standing rule — no sidecar files in data folders. The manifest violates it. The exception was accepted deliberately, with both sides recorded in the decision log. Good architecture is not rule-free; it is exceptions-with-reasons.

Build a reconciliation report for the migration. For each object x pull
generation: validate against the contract (filter-aware counts via file
footers, schema vs field list). Output the damage map, classify each
damaged object into the 5-class cheapest-first triage, and list class-4
cases separately with their diagnosis for human follow-up. Cite the
manifest for every verdict.

Then — and only then — cut over

The sequence, assembled from all three lessons, per object: reconcile → remediate cheapest-first → re-reconcile → enable the incremental schedule (it reads watermark = T0 from the control row, pulls > T0; nothing in its config changes) → decommission the old feed. Per-object, never big-bang: WORK_ORDERS can be live in the lake while METER_READINGS finishes class-2 remediation. The migration’s own delta mode stays available as the bridge if any object needs to fall back.

Key takeaway

Validate each plane against the contract, not systems against systems; make it affordable with footer reads and filter-aware counts; ask “does ANY pull satisfy?” and triage the residue cheapest-first, routing contract disputes to humans; let manifests carry provenance so nothing depends on memory. Then cut over per object, on evidence. Module 3 is done — what remains is making the applications production-worthy.