Parity for Batch Jobs
Updated
IntermediateWhat you'll learn
~55 min- Compare a legacy job's output file with the rebuilt job's by business key (a record's identifying column, here CaseId) instead of by line, reporting missing, extra and field-level differences under tolerances written in a file
- Hold the export to its positional contract (consumers read column 3, not a column named Status) so an added or swapped column fails before a consumer reads the wrong value
- Decode a mainframe-origin input file strictly: EBCDIC (IBM's character encoding) code page 037 text and COMP-3 packed decimal (two digits per byte, sign in the last half-byte), failing on a malformed nibble
- Make a re-run or a restart of the same business date deliver one identical file, using a run ledger (a record of which dates have been built and delivered) and a destination that creates the file only if it does not already exist
Lesson 24’s dependency map named Intake_Nightly_Export, a SQL Agent job (SQL Server’s built-in scheduler) with a CmdExec step (it runs an operating-system command) that writes four columns of dbo.Intake to a positional CSV, and left it as U-08: nobody owns it, and the target database cannot run its step. The golden cases of lessons 22 to 25 compared one request with one response. A batch job has no request. It reads a table or a file at a scheduled time, writes a file, and somebody downstream picks that file up. Parity here means three things the earlier harness never checked: the file is the same record for record, the file is delivered once however the job is started, and the job’s input is decoded without guessing.
Before the detail — Artifact: src/modernization/batch/ (a keyed record diff over a real CSV record parser, strict EBCDIC and COMP-3 decoders, a run ledger with a conditional-create destination), a tolerances-and-contract file fixtures/batch/export-contract.json, and tests/unit/batch.test.ts. Acceptance test: a new file with the same records in a different order passes, a new file with an added or swapped column fails, a malformed sign nibble fails naming its record and field, an unclosed quote is reported as a malformed file, and two deliveries of the same business date started at the same moment leave one file. Status: reusable guidance; the keyed diff and the ledger are this module’s, built from published practice (idempotence, expand and contract), and Fowler’s legacy-displacement hub lists parallel run as a delivery strategy without a pattern page of its own.
| Problem | The pattern | Where your platform lesson lands it | You decide |
|---|---|---|---|
| Which record is which | A keyed diff: match legacy and new records on the business key, order-insensitive unless the contract says order matters (the alternative: sort both files and compare line by line, which reports one early difference as a hundred and hides a duplicated key) | Unchanged on every platform | Whether any consumer relies on row order (U-50) |
| Consumers read by position | The positional contract is the column list in a file the test reads; an added or swapped column fails by position (the alternative: add a header row and move every consumer to names, which is a change to every consumer) | Unchanged | Whether consumers can move to names, and who tells them |
| Where the two may differ | Tolerances declared in a file, each with its reason (the alternative: normalize both files before comparing, which hides the difference instead of declaring it) | Unchanged | Who signs a tolerance |
| Mainframe-origin input | Decode strictly and refuse what means nothing (the alternative: ask the sending team for a converted file, and still keep the check) | Unchanged | Which code page the sender used (U-52) |
| Run once, restart safely | A run ledger keyed by business date, and a destination that creates the file atomically and only if absent, so it refuses a different file for a delivered date even when two runs start together (the alternative: a scheduler’s own skip-if-done setting, which covers neither a crash between delivery and the record of it nor two starts at once) | 34: Agent on Managed Instance keeps T-SQL steps and not CmdExec, so the rebuilt job is scheduled by Functions or Logic Apps · 35: a container task on Fargate, Step Functions when the job has steps (GovCloud-West: FIPS endpoints only, no HTTPS API calls) · 36: Cloud Scheduler starting a Cloud Run workload, Workflows when the job has steps | Which scheduler your agency has authorized (status box) |
What the old job does is the specification
Lesson 22’s rule applies unchanged: the legacy system supplies the expected outputs. For a batch job, the expected output is a file the old job wrote on the restore, and the agent’s part is to propose the input. Its candidates here are input tables or input files: a case that straddles midnight, a row with a trailing blank in CaseType, a Status with a comma in it, two cases submitted in the same second. The pipeline’s bench runner (lesson 25) runs the old job on each candidate and records what it wrote. The agent never writes a line of an expected file.
Plan the change with your agent
Write down what the job does before you rebuild it. Open your agent in the project root and switch to Plan mode (Shift+Tab, /plan, or the mode picker) so the first turns only read:
Read modernization/db-dependents.md and our lesson 24 msdb job-step notes, and plan modernization/batch/job-notes.md for Intake_Nightly_Export plus candidate input tables for the bench runner. Ask me anything unclear before writing files.What to check in the plan before you approve
modernization/batch/job-notes.mdrecords only what evidence shows: the query, business-date selection and time zone, output filename, positional column order, delimiter and quoting, delivery target, and step retry behavior.- Anything not visible in the repository evidence (who consumes the file and whether row order matters, what a second run of the legacy job does, Which EBCDIC code page the mainframe feed uses, and how legacy handles commas or quotes in
Status) goes intounknowns.md(U-50..U-53) without guessing. - Candidate cases (
modernization/batch/candidates/) contain synthetic input tables only (midnight day boundaries, quoting edge cases, trailing blanks, duplicate timestamps), leaving expected outputs for the bench runner to record from the legacy copy.
If the agent’s plan offers to generate the target CSV rows for the candidate cases, correct it before approving:
Write only the candidate input tables in modernization/batch/candidates/. Do not generate expected CSV output files; the bench runner records those from the legacy copy.Seven entries come back: three rules the new job must honor, three unknowns, and one decision the new job makes for itself:
| ID | Entry |
|---|---|
| R-50 | The file has no header and four columns in this order: CaseId, CaseType, Status, SubmittedAt. Consumers read by position (lesson 24, msdb.dbo.sysjobsteps) |
| R-51 | One file per business date, named for that date; the date selects rows whose SubmittedAt falls on that calendar day |
| R-52 | A second run for a delivered date with the same source rows delivers nothing new. A second run with different source rows for that date is not a re-run: it fails with ConflictingRedelivery and an owner rules on it (see “Run once, restart safely”) |
| D-50 | The new job writes a field that holds a comma, a double quote or a line break inside double quotes, with a quote doubled (RFC 4180 minimal quoting). This is a decision, not parity, until U-53 is answered |
| U-50 | Who consumes the file and whether any consumer relies on row order. Ask the owner of the receiving folder |
| U-51 | What the old job’s second run does to the file on the restore: overwrite, append or fail. Record it with the bench runner before R-52 is final |
| U-52 | Which code page the mainframe feed uses (037 or another EBCDIC page). Ask the sending team; the decoder below assumes 037 |
| U-53 | How the old job writes a Status that holds a comma, a quote or a line break: quoted, unquoted, or never occurs. The candidate cases include a comma in Status, so the bench runner records the answer. The old query’s shape in the env test below joins columns with CONCAT and does not quote |
R-52 is the rule the new job chooses, not one it inherits. U-51 is what the old job did, and until it is recorded the new job’s behavior is stated as a decision, not as parity. D-50 is the same kind of statement for quoting: if U-53 shows the old job never quotes, a comma in Status makes the old file unreadable by position, and the right ruling may be to keep the old behavior for consumers that split on commas or to move them. That is the owner’s call (U-50), and the contract records which one was made.
The keyed record diff
A file diff compares lines. A parity check compares records, and the record is named by its key. Two files with the same records in a different order are the same output unless somebody downstream reads order, and a single missing record should be reported once, by key, not as the hundred lines that shifted. The contract in fixtures/batch/export-contract.json is data, so a reviewer can read it and a code owner (lesson 25) can protect it (reference excerpt from gov-dev-reference-app; compare with what your agent writes):
// fixtures/batch/export-contract.json (excerpt) "key": "CaseId", "columns": ["CaseId", "CaseType", "Status", "SubmittedAt"], "ordered": false, "quoting": { "rule": "rfc4180-minimal", "status": "the new job's decision D-50, not parity: how the legacy export writes a comma, quote or line break is unrecorded (U-53)" }, "tolerances": [ { "column": "SubmittedAt", "kind": "timestamp", "withinMs": 1, "reason": "the legacy job writes datetime2 text with seven fractional digits, the new job writes ISO-8601 UTC to the millisecond; the same instant to the millisecond is equal" } ]columns is the positional contract (R-50): the diff reads the n-th field as the n-th name. ordered is false until U-50 says otherwise. quoting records D-50 and its status, so the contract itself says that quoting is a decision and not yet parity (U-53). The only tolerance is the timestamp format, with its reason on the line, as in lesson 23: an unexplained tolerance is a way to turn a red run green. The types and the diff itself:
REFERENCE CODE src/modernization/batch/keyed-diff.ts (excerpt): Contract types and diffFiles() open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// src/modernization/batch/keyed-diff.ts (excerpt)export type Tolerance = | { column: string; kind: 'timestamp'; withinMs: number; reason: string } | { column: string; kind: 'numeric'; abs: number; reason: string } | { column: string; kind: 'text'; fold: Array<'trim' | 'case'>; reason: string };
export interface Contract { key: string; // The file has no header. Consumers read by position, so this list IS the contract. columns: string[]; // True only when row order is part of what a consumer relies on. ordered: boolean; // How a field holding a comma, a quote or a line break is written. The reader accepts RFC 4180 // quoting only; this records what the renderer does and whether the legacy job is known to match. quoting?: { rule: 'rfc4180-minimal'; status: string }; tolerances: Tolerance[];}// ...export function diffFiles(legacyText: string, newText: string, contract: Contract): Finding[] { validateContract(contract); const findings: Finding[] = []; const legacy = parseRows(legacyText, contract, 'legacy', findings); const next = parseRows(newText, contract, 'new', findings); // A file that stops being readable has no complete record set to compare: report that alone. if (legacy.malformed || next.malformed) return findings; const keyAt = contract.columns.indexOf(contract.key); const tolerance = new Map(contract.tolerances.map((t) => [t.column, t])); const badKeys = new Set([...legacy.badKeys, ...next.badKeys]);
const index = (rows: string[][], side: 'legacy' | 'new') => { const byKey = new Map<string, string[]>(); for (const r of rows) { if (byKey.has(r[keyAt])) findings.push({ kind: 'duplicate-key', side, key: r[keyAt] }); else byKey.set(r[keyAt], r); } return byKey; }; const legacyByKey = index(legacy.rows, 'legacy'); const newByKey = index(next.rows, 'new');
for (const [key, l] of legacyByKey) { const n = newByKey.get(key); if (!n) { if (!badKeys.has(key)) findings.push({ kind: 'missing', key }); continue; } contract.columns.forEach((column, c) => { if (!equalUnder(tolerance.get(column), l[c], n[c])) findings.push({ kind: 'field', key, column, legacy: l[c], new: n[c] }); }); } for (const key of newByKey.keys()) if (!legacyByKey.has(key) && !badKeys.has(key)) findings.push({ kind: 'extra', key });
if (contract.ordered) { // Order of the keys both files hold; a missing or extra key is already its own finding. const a = [...legacyByKey.keys()].filter((k) => newByKey.has(k)); const b = [...newByKey.keys()].filter((k) => legacyByKey.has(k)); const at = a.findIndex((k, i) => k !== b[i]); if (at >= 0) findings.push({ kind: 'order', position: at + 1, legacy: a[at], new: b[at] }); } return findings;}Five details carry the weight.
The reader is a record parser, not a split on newlines. A first version split the file on line breaks and then looked for quotes, which cuts a quoted field that holds a line break in two, and accepts an unclosed quote: legacy 1,abc against new 1,"abc returned no findings at all. The parser reads character by character in RFC 4180 style: a field that opens with a quote runs to the matching quote, "" inside it is one quote, and commas and line breaks inside it are data. Anything the RFC does not allow is reported, not repaired:
REFERENCE CODE src/modernization/batch/keyed-diff.ts (excerpt): RFC 4180 parseCsv() record reader open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// src/modernization/batch/keyed-diff.ts (excerpt)export function parseCsv(text: string): { records: CsvRecord[]; malformed?: { line: number; detail: string } } { const records: CsvRecord[] = []; let fields: string[] = []; let cur = ''; let line = 1; let start = 1; let state: 'start' | 'plain' | 'quoted' | 'after' = 'start'; let sawQuote = false; const endRecord = () => { fields.push(cur); if (!(fields.length === 1 && fields[0] === '' && !sawQuote)) records.push({ line: start, fields }); fields = []; cur = ''; sawQuote = false; }; for (let i = text.charCodeAt(0) === 0xfeff ? 1 : 0; i < text.length; i++) { // a leading UTF-8 byte-order mark is not data const c = text[i]; if (state === 'quoted') { if (c === '"' && text[i + 1] === '"') { cur += '"'; i++; } else if (c === '"') state = 'after'; else { if (c === '\n') line++; cur += c; } continue; } if (c === '\r' && text[i + 1] === '\n') continue; if (c === '\r') return { records, malformed: { line, detail: `bare carriage return outside a quoted field (record starting at line ${start})` } }; if (c === '\n') { endRecord(); line++; start = line; state = 'start'; continue; } if (c === ',') { fields.push(cur); cur = ''; state = 'start'; continue; } if (state === 'after') return { records, malformed: { line, detail: `text after the closing quote of a field (record starting at line ${start})` } }; if (c === '"') { if (state !== 'start') return { records, malformed: { line, detail: `quote inside an unquoted field (record starting at line ${start})` } }; state = 'quoted'; sawQuote = true; continue; } cur += c; state = 'plain'; } if (state === 'quoted') return { records, malformed: { line: start, detail: 'unclosed quote: the file ends inside a quoted field' } }; if (state !== 'start' || fields.length > 0 || cur !== '') endRecord(); return { records };}An unclosed quote, a quote inside an unquoted field and text after a closing quote each become a malformed finding that names the line the record starts on. A file that cannot be read has no complete record set, so diffFiles returns the malformed finding alone and compares nothing else; a report that also listed forty “missing” keys would point away from the real fault.
A bad row is one finding. A row with the wrong number of fields is a shape finding before any value is compared, which is how an added column fails. Its key is remembered so the same row is not reported a second time as missing or extra:
REFERENCE CODE src/modernization/batch/keyed-diff.ts (excerpt): parseRows() shape validation open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// src/modernization/batch/keyed-diff.ts (excerpt)export function parseRows(text: string, contract: Contract, side: 'legacy' | 'new', findings: Finding[]): { rows: string[][]; badKeys: Set<string>; malformed: boolean } { const keyAt = contract.columns.indexOf(contract.key); const rows: string[][] = []; const badKeys = new Set<string>(); const { records, malformed } = parseCsv(text); for (const r of records) { if (r.fields.length !== contract.columns.length) { findings.push({ kind: 'shape', side, line: r.line, detail: `${r.fields.length} fields, contract has ${contract.columns.length}` }); if (keyAt < r.fields.length) badKeys.add(r.fields[keyAt]); } else { rows.push(r.fields); } } if (malformed) findings.push({ kind: 'malformed', side, line: malformed.line, detail: malformed.detail }); return { rows, badKeys, malformed: !!malformed };}The contract is checked before the files. validateContract refuses a key or a tolerance column that is not one of the columns, because indexOf of a missing name is -1 and a diff that reads field -1 compares nothing and passes.
Order is checked on the keys both files hold. Comparing positions only when the two files have the same number of rows skipped the order check exactly when something was already wrong. The check now filters each side to the shared keys, so one missing record is one missing finding and does not hide a reordering of the rest.
A timestamp must be a real instant. Date.UTC rolls February 30 over to March 2, which made a legacy 2026-02-30 equal a new 2026-03-02 under the timestamp tolerance. The reference parser in src/modernization/batch/keyed-diff.ts from gov-dev-reference-app keeps the instant only if the date and the clock read back the same:
// src/modernization/batch/keyed-diff.ts (excerpt)const instant = (s: string): number | null => { const m = /^(\d{4})-(\d{2})-(\d{2})[T ](\d{2}):(\d{2}):(\d{2})(?:\.(\d+))?Z?$/.exec(s.trim()); if (!m) return null; const [y, mo, d, h, mi, se] = [+m[1], +m[2], +m[3], +m[4], +m[5], +m[6]]; const ms = Number((m[7] ?? '0').padEnd(3, '0').slice(0, 3)); const t = Date.UTC(y, mo - 1, d, h, mi, se, ms); const back = new Date(t); if (back.getUTCFullYear() !== y || back.getUTCMonth() !== mo - 1 || back.getUTCDate() !== d || h > 23 || mi > 59 || se > 59) return null; return t;};A key that appears twice is reported, not collapsed into the later row, and missing, extra and field each name the key, so the report reads “10233 is missing” and not “line 3 differs”.
Mainframe-origin input: decode, and refuse
If the batch’s input is a file from a mainframe, two encodings stand between the file and your code. Text is EBCDIC, and the common variant for US English is code page 037. Numbers are often COMP-3, COBOL’s packed decimal: each byte holds two decimal digits, and the last half-byte (nibble) is the sign. C is positive, D is negative and F is positive with no sign (unsigned). A reader that treats those bytes as ASCII, or as hex digits, produces numbers that look plausible, which is the worst failure. The decoder is strict, and says why it stopped:
REFERENCE CODE src/modernization/batch/decode.ts (excerpt): strict COMP-3 packed-decimal decoder open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// src/modernization/batch/decode.ts (excerpt)// COMP-3: two decimal digits per byte, the last nibble is the sign. C and F are positive// (F means unsigned), D is negative. Other sign nibbles are refused, not guessed.// With `digits` (the layout's declared digit count) the field must be exactly the bytes that// count needs, and when the count is even the leading nibble is padding and must be 0.// Negative zero (a D sign on a value of zero) is accepted and decoded as plain zero, so// -0 and 0 are the same value downstream; encodeComp3 writes zero with a C sign to match.export function decodeComp3(bytes: Uint8Array, scale = 0, digits?: number): string { if (bytes.length === 0) throw new DecodeError('empty COMP-3 field'); if (digits !== undefined && bytes.length !== Math.floor(digits / 2) + 1) { throw new DecodeError(`${digits} digits need ${Math.floor(digits / 2) + 1} bytes, field has ${bytes.length}`); } let digitText = ''; for (let i = 0; i < bytes.length; i++) { const hi = bytes[i] >> 4; const lo = bytes[i] & 0x0f; if (hi > 9) throw new DecodeError(`byte ${i}: high nibble 0x${hi.toString(16).toUpperCase()} is not a digit`); digitText += hi; if (i < bytes.length - 1) { if (lo > 9) throw new DecodeError(`byte ${i}: low nibble 0x${lo.toString(16).toUpperCase()} is not a digit`); digitText += lo; } } const sign = bytes[bytes.length - 1] & 0x0f; if (sign !== 0xc && sign !== 0xd && sign !== 0xf) { throw new DecodeError(`sign nibble 0x${sign.toString(16).toUpperCase()} is not C, D or F`); } if (digits !== undefined && digitText.length > digits && digitText[0] !== '0') { throw new DecodeError(`${digits} digits declared, but the padding nibble is 0x${digitText[0]} and the value does not fit`); } digitText = digitText.padStart(scale + 1, '0'); const whole = digitText.slice(0, digitText.length - scale).replace(/^0+(?=\d)/, ''); const text = scale > 0 ? `${whole}.${digitText.slice(digitText.length - scale)}` : whole; return sign === 0xd && /[1-9]/.test(text) ? `-${text}` : text;}A sign nibble outside C, D and F throws; so does a digit nibble above 9. Some sources also treat A and E as positive and B as negative; this decoder refuses them, so the first real file that carries one stops the run and a person decides, and the decision goes in unknowns.md as U-52’s neighbor. The text decoder is a 256-entry table generated from a reference codec and checked against it, because code pages 037 and 1047 differ on a few punctuation bytes and a wrong table corrupts them silently.
Three rules came from review of an earlier version, each with a test that failed against it. The declared digit count is enforced. A layout field with digits: 2 is two bytes; the first of its three digit nibbles is padding and must be 0, so 12 3C is refused (it holds 123, which does not fit two digits) while 01 2C is 12. A field of the wrong byte length is refused too. Negative zero is decided, not left to chance. A D sign on a value of zero decodes as plain zero, and the encoder writes zero with a C sign, so -0 and 0 are one value on both sides of the round trip. Trimming is a choice the layout makes per field. A fixed-width text field is padded with spaces, and a trailing blank can be data (the candidate cases include a trailing blank in CaseType), so the decoder keeps the field exactly as written unless the layout says "trim": true, and a layout that does not say either way is refused:
REFERENCE CONFIG fixtures/batch/referral-feed.layout.json (excerpt): mainframe referral feed field layout open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// fixtures/batch/referral-feed.layout.json (excerpt)[ { "name": "CaseId", "type": "text", "length": 8, "trim": true }, { "name": "CaseType", "type": "text", "length": 10, "trim": true }, { "name": "HouseholdSize", "type": "comp3", "digits": 3 }, { "name": "BenefitAmount", "type": "comp3", "digits": 7, "scale": 2 }]Here is the reference FieldSpec union and field decoding dispatch in src/modernization/batch/decode.ts from gov-dev-reference-app:
// src/modernization/batch/decode.ts (excerpt)export type FieldSpec = // `trim` is explicit: true drops trailing spaces (the EBCDIC pad), false keeps the field exactly // as written, because a trailing blank can be data (a parity case the lesson lists). | { name: string; type: 'text'; length: number; trim: boolean } | { name: string; type: 'comp3'; digits: number; scale?: number };// ... row[f.name] = f.type === 'text' ? (f.trim ? decodeEbcdic037(raw).replace(/ +$/, '') : decodeEbcdic037(raw)) : decodeComp3(raw, f.scale ?? 0, f.digits);A bad field produces this error output naming its record and field:
record 2, field HouseholdSize: sign nibble 0xA is not C, D or FAmounts stay strings ("1250.00", "-75.50"), never floats, so a tolerance on a money column is a decision someone wrote down.
Run once, restart safely
The new job may be started by the scheduler twice, by an operator who is not sure the first run worked, by a restart after a crash, or by two schedulers at the same moment. Parity with the old job includes what happens then. The mechanism has three parts: the file for a business date is a pure function of the rows, so a re-run produces identical bytes; a ledger records built and delivered by date; and the destination creates a file only if the name is free, so it refuses a different file under a name it already holds.
The destination is where two simultaneous runs are decided, so it does not check and then write. A first version asked whether the file existed and then renamed a shared .part file into place; two runs that both asked before either renamed both delivered, and both wrote the same temp name. The version below writes the bytes to a temp file named for this call alone, then makes the destination name appear with one linkSync, which fails with EEXIST when the name is taken. The loser reads the winner’s file and either reports unchanged (same bytes) or throws ConflictingRedelivery (different bytes). The winner’s file is never touched:
REFERENCE CODE src/modernization/batch/run-ledger.ts (excerpt): atomic conditional-create directorySink() open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// src/modernization/batch/run-ledger.ts (excerpt)// A destination folder that refuses to overwrite a delivered file with different bytes.// The create is atomic and conditional: the bytes go to a temp file that is unique to this call,// then linkSync(temp, dest) makes dest appear complete or fails with EEXIST. There is no// separate "does it exist" check for another process to slip between. A production sink also// fsyncs the temp file before the link and the directory after it, so a power cut cannot leave// a delivered name with unwritten bytes; this demo omits both. (linkSync needs a local// POSIX-style filesystem. On an object store the same step is a conditional put, If-None-Match.)export function directorySink(dir: string, seam: { beforeLink?: (name: string) => void } = {}): DeliverySink { mkdirSync(dir, { recursive: true }); return { put(name, bytes) { const dest = join(dir, name); const tmp = join(dir, `.${name}.${process.pid}.${randomUUID()}.part`); writeFileSync(tmp, bytes); try { seam.beforeLink?.(name); linkSync(tmp, dest); return 'created'; } catch (e) { if ((e as NodeJS.ErrnoException).code !== 'EEXIST') throw e; if (sha(readFileSync(dest)) === sha(bytes)) return 'unchanged'; throw new ConflictingRedelivery(`${name} was already delivered with different content`); } finally { unlinkSync(tmp); } }, };}The demo omits one production step. Before the link, a production sink calls fsync on the temp file (and on the directory after the link), so a power cut cannot leave a delivered name that points at bytes not yet on disk. Without it, a crash can make the destination hold a file that the ledger and the loser both believe is complete. The link needs a local POSIX-style filesystem; on an object store the same step is a conditional put that fails if the key exists, and which service offers it on your platform is a question for lessons 34 to 36, not something this lesson sources.
Rows that hold a comma, a quote or a line break are written the way the reader parses them (D-50), and the reference csvField helper in gov-dev-reference-app puts that choice in one place:
// src/modernization/batch/run-ledger.ts (excerpt)// CSV quoting, decision D-50 (the legacy job's own behavior is unrecorded, U-53): a field is// wrapped in double quotes only when it holds a comma, a double quote, CR or LF, and a quote// inside it is doubled (RFC 4180). The reader in keyed-diff.ts parses exactly this.const csvField = (v: string | number) => { const s = String(v); return /[",\r\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;};The run itself:
REFERENCE CODE src/modernization/batch/run-ledger.ts (excerpt): runExport() state machine and crash recovery open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// src/modernization/batch/run-ledger.ts (excerpt)export function runExport(o: RunOptions): { status: 'delivered' | 'already-delivered'; sha256: string } { mkdirSync(o.workDir, { recursive: true }); const ledgerPath = join(o.workDir, 'run-ledger.json'); const ledger: Ledger = existsSync(ledgerPath) ? JSON.parse(readFileSync(ledgerPath, 'utf8')) : {}; const save = () => saveAtomically(ledgerPath, JSON.stringify(ledger, null, 2)); const name = `intake-export-${o.businessDate}.csv`; const path = join(o.workDir, name);
const done = ledger[o.businessDate]; if (done?.state === 'delivered') return { status: 'already-delivered', sha256: done.sha256 };
// Reuse the file a crashed run built, if it is intact; otherwise build it again. let bytes: Buffer; if (done?.state === 'built' && existsSync(path) && sha(readFileSync(path)) === done.sha256) { bytes = readFileSync(path); } else { bytes = renderExport(o.rowsFor(o.businessDate)); saveAtomically(path, bytes); ledger[o.businessDate] = { state: 'built', sha256: sha(bytes), file: name }; save(); } if (o.crashAt === 'after-build') throw new Error('simulated crash after build');
o.sink.put(name, bytes); // 'unchanged' means a crashed run already delivered it if (o.crashAt === 'after-put') throw new Error('simulated crash after put'); ledger[o.businessDate] = { state: 'delivered', sha256: sha(bytes), file: name }; save(); return { status: 'delivered', sha256: sha(bytes) };Read it as a restart table. Crash before the build: nothing recorded, so the next run builds. Crash after the build (after-build in the tests): the ledger says built, the file is intact by hash, and the restart delivers that exact file. Crash after the delivery but before the ledger update (after-put): the restart delivers again, the destination answers unchanged, and nothing is duplicated. Ledger says delivered: the run returns at once. Two runs at once on the same business date: both may build and both may reach the destination; the conditional create lets exactly one create the file, and the ledger, which is not a lock, may be written twice with the same content. The ledger is a file here; in your job it lives where a restart on a different machine can still read it, which is a database table or the object store the job already writes to.
What “idempotent” covers here. A re-run with the same source rows for a delivered business date delivers nothing new. A re-run with changed source rows for that date is an error, not a re-run: when the ledger no longer says delivered (it was lost, or a crash left it at built), the new bytes meet the old file and the destination throws ConflictingRedelivery. When the ledger does say delivered, the run returns already-delivered without reading the rows at all, so a late-arriving row is not delivered and is not reported either. If a late row matters, that is a rule for the owner (U-50), for example a correction file under a new name, and not something the ledger can infer.
The old job’s own behavior on a second run is U-51. Record it on the restore, then choose: if the old job overwrote the file, the new job’s refusal is stricter, and that difference goes on lesson 25’s ruling list as a decision with a named person, not as a silent improvement.
Run windows and where the job runs
A run window is the time between the moment the business date closes and the moment a consumer expects the file. Three things have to match the old job: which calendar day a row belongs to and in which time zone (the env test below has a row at 23:59:59.999 and one at 00:00:00 to pin the boundary), what time the job starts, and how long it may take. The old job’s start time and its retry setting sit in the msdb job tables lesson 24 already read.
The scheduler is where platforms differ. On Azure SQL Managed Instance the Agent runs, but Microsoft’s differences page says “Command shell isn’t yet supported” (T-SQL differences), so the CmdExec step cannot move as it is. That is why the export is rebuilt as a program with a ledger, and the Agent job, if you keep one, only starts it. Lesson 34 lands the schedule on the Azure services in the status box; lesson 35 on a container task, with Step Functions when the job has several steps and its GovCloud-West constraint; lesson 36 on Cloud Scheduler and Cloud Run. Temporal in a gov region is not sourced, so self-host it or leave it out. Whichever starts the job, the ledger is what makes a double start harmless.
For several jobs, run discovery as one subagent per job (lesson 22), each writing its own job-notes.md, and rebuild them in parallel in separate worktrees (Module 19 lesson 5: subagent, then background, then worktree ladder, at Subagents and Orchestration). Each rebuild is gated by the same referee files: the contract, the tolerances and the tests are code-owned, and no agent edits them.
Watch it work
- Typed in the terminal:
claude - Mode:Plan mode (Shift+Tab)
- You type in the session: Read modernization/db-dependents.md and write modernization/batch/job-notes.md for Intake_Nightly_Export from evidence only. Then propose candidate input tables. No expected outputs.
- The tool replies: Read modernization/db-dependents.md, the msdb job-step text from lesson 24, modernization/unknowns.md. Wrote modernization/batch/job-notes.md. From evidence: four columns CaseId, CaseType, Status, SubmittedAt; positional CSV; CmdExec step. Not shown by anything I can read: the time zone for the business date, what a second run does to the file, who reads the file. Added U-50, U-51; the mainframe feed's code page is U-52. Proposed 10 candidate input tables in modernization/batch/candidates/ (midnight boundary, a trailing blank in CaseType, a comma in Status, two cases in the same second). They contain inputs only; the bench runner records the legacy outputs. Plan: contract file, keyed diff, decoders, run ledger, tests. Approve to proceed.
- You type in the session: Approved. Implement, then run the unit tests.
- The tool replies: Wrote src/modernization/batch/{keyed-diff,decode,run-ledger,deliver-once}.ts, fixtures/batch/export-contract.json, tests/unit/batch.test.ts. npx vitest run tests/unit/batch.test.ts Test Files 1 passed (1) Tests 39 passed (39)
- You type in the session: Plant a fault: match records by line position instead of by key, and run again.
- The tool replies: Changed one line in keyed-diff.ts: const n = next.rows[legacy.rows.indexOf(l)]; npx vitest run tests/unit/batch.test.ts × a reordered file matches: order is not part of the contract × a reordered file fails when a consumer depends on order × reports missing, extra and field differences by key × a quoted field with a comma, a doubled quote and a line break is one field of one record × order is checked on the keys both files hold, even when the row counts differ Test Files 1 failed (1) Tests 5 failed | 34 passed (39) The same five records in reverse order now report CaseId and SubmittedAt differences on four of the five rows. The unordered case is the one that matters: a correct export that merely sorts differently fails the build.
- You type in the session: Restore the key lookup and re-run.
- The tool replies: Restored const n = newByKey.get(key); npx vitest run tests/unit/batch.test.ts Test Files 1 passed (1) Tests 39 passed (39) The expected outputs are still the files the bench runner recorded. I did not write any.
Illustrative session; exact output varies by tool and agency configuration.
The test output above comes from running this code. The reference’s legacy-export-20261104.csv is a stand-in written in the format the old query produces (the env test below runs that query’s shape on a real SQL Server); with your own restore, the bench runner records the real file and you replace the stand-in. Nothing here ran a SQL Agent job or a mainframe.
Prove it
From gov-dev-reference-app, run these in order.
1. The suite. npm test runs tests/unit/batch.test.ts among the others. On its own: npx vitest run tests/unit/batch.test.ts reports 39 passed in seven groups: code page 037 (2), COMP-3 (6), the layout file (3), the feed (2), the keyed diff (14), re-delivery (10), parser edges (2: a byte-order mark, a bare carriage return). Each review fix came with a test that failed against the earlier code: 17 of the 39 fail on it, among them the unclosed quote, the quoted line break, the doubled finding for a bad row, the order check on unequal lengths, the declared digit count and the two-process delivery.
2. The sign nibbles and the digit count. The COMP-3 group reads 12 3C as 123, 12 3D as -123 and 12 3F as 123; it reads 01 25 00 0C with scale 2 as 1250.00 and 00 07 55 0D as -75.50. It throws on 12 3A, 12 35 (sign nibbles), 1A 3C (a low nibble that is not a digit), B2 3C (a high nibble that is not a digit) and an empty field. With a declared digit count it refuses 12 3C at 2 digits and accepts 01 2C as 12; it encodes -0.00 as 00 00 00 0C and decodes a D on zero as 0.00.
3. The plant. Change const n = newByKey.get(key); in keyed-diff.ts to const n = next.rows[legacy.rows.indexOf(l)];. Expected: 5 failed, 34 passed, the failing tests being the reordered-file pair, the missing, extra and field report, the quoted-field test and the shared-key order test. Restore the line; 37 pass.
4. A real database. The env test creates its own database batch with six dbo.Intake rows, runs the old query’s shape (CONVERT(varchar(27), SubmittedAt, 121), seven fractional digits) and the new job’s typed read, and diffs the two files under the contract. With the container up (npm run local:up), M14_ENV=1 npx vitest run tests/env/batch.env.test.ts passes 3 tests: the files agree for 2026-11-04, 2026-11-05 holds only the midnight row, and a status of ON HOLD, REVIEW on 2026-11-06 fails the contract by shape on the legacy side, because the old query’s shape joins with CONCAT and does not quote (U-53 asks whether the real job does).
5. Restart and collision. In the re-delivery group, “run twice” ends with one entry in the delivery log; “crash after the build, restart” yields the same bytes; “crash after the put but before the ledger” ends with the destination holding one file and created once; a lost ledger with changed rows throws ConflictingRedelivery. Two tests start two operating-system processes at the same instant (deliver-once.ts, 25 business dates each, into one folder). With identical content, each date has exactly one created and the folder holds byte-identical files; with different content, each date has one created and one conflict, and the file is the winner’s, untouched. A third test forces the loss inside one process, by delivering the same name between a call’s temp write and its link.
Availability, authorization, agency approval, region support, and price are five separate facts. This box holds the dated service facts for this lesson; the prose states none of them. Managed Instance: SQL Agent is supported and “Command shell isn’t yet supported” (T-SQL differences, page built 2026-10-19); Agent jobs and logins are instance-level objects that a database backup does not carry, and the Database Migration Service moves Agent job steps of the T-SQL type only. Azure Logic Apps and Azure Functions appear in the Azure Government service table with FedRAMP High, DoD IL2, IL4 and IL5 with Wide Area (IL5WI) scope (audit scope, updated 2026-09-21); Durable Functions has no row of its own and whether every Durable backend is offered there is UNVERIFIED. AWS: Step Functions is offered in both GovCloud regions; “US GovCloud West only supports FIPS endpoints” and “Support to call HTTPS APIs is not available” (GovCloud Step Functions); ECS and Fargate are offered in both regions (GovCloud ECS); whether RDS for SQL Server runs Agent jobs, and the AWS service that would start a Fargate task on a schedule, are not covered by the sources read: UNVERIFIED. Google: the Assured Workloads supported-products page lists Workflows, Cloud Run, Cloud SQL and Cloud Scheduler under FedRAMP High (supported products; only the first 100,000 characters of that page were read, so IL4 and IL5 are UNVERIFIED). Cloud SQL for SQL Server lists “SQL Server Agent enabled to facilitate replication and other jobs” (features). Temporal in a gov region: UNVERIFIED. Parallel run is named as a delivery strategy, without a definition, on Fowler’s Patterns of Legacy Displacement hub (2024-03-05). The code page 037 and COMP-3 conventions in this lesson are the author’s reading of widely documented IBM formats and were not checked against a fetched IBM page: check them against your sender’s layout. Recheck every cited source and your agency’s authorization boundary before you rely on any of it.
The rebuilt export writes the same 5,000 records as the old one, sorted by CaseId; the old file was in insertion order. A line-by-line diff reports 4,900 differences. What does the parity check do?
The decoder meets a sign nibble of 0xA in a COMP-3 field of a real file. A teammate says A is positive in some systems, so treat it as C. What is right?
Practice status — batch job parity and safe re-delivery
| Practice | Status | Also called |
|---|---|---|
| keyed record diff, order-insensitive unless the contract says otherwise | strong optional | record-level reconciliation; keyed compare |
| positional column contract checked by a test | strong optional | schema contract for files; consumer-driven contract |
| tolerances in a file, each with a reason | strong optional | declared differences; known-difference list |
| strict decoding of EBCDIC and packed decimal | common baseline | copybook-driven conversion; fail-fast parsing |
| run ledger keyed by business date, delivered through an atomic create-if-absent | common baseline | idempotent job; exactly-once effect; checkpoint and restart; conditional put |
| CSV fields quoted and parsed under RFC 4180 rules | common baseline | quoted fields; CSV escaping |
| parallel run of old and new before cutover | emerging | shadow run; parallel running |
Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging
Key takeaway
A batch job has no request to replay, so its parity is a file: the same records by key, in the contract’s column positions, within tolerances written down with their reasons, delivered once however the scheduler starts it. The old job writes the expected file; the agent proposes the inputs and nothing else. A mainframe feed is decoded strictly, because a decoder that guesses turns a bad nibble into a plausible number. The scheduler differs by platform and the ledger does not. Lesson 31 takes what happens to a case after intake: its states, assignments and timers.
- LEADERSHIP DECISION
- the export's owner and consumers are named (U-50) before cutover, and any difference from the old job's re-run behavior is a ruling, not a quiet improvement
- PRACTITIONER ACTION
- put the contract, the tolerances and the batch tests under code owners, record the old job's output on the restore, and run the keyed diff and the restart tests in every build
- SUCCESS MEASURE
- contract or revenue risk avoided: zero unruled differences between the two files for a full run window, and one delivery per business date across forced re-runs
Legend: LEADERSHIP DECISION = what a leader decides · PRACTITIONER ACTION = what the engineer does next · SUCCESS MEASURE = how you know, in calendar time, assessor-hours or cost, contract or revenue risk, or audit finding avoided