What the Agent Got Wrong

Why this needs a name

You have a definition, an acceptance test, a reconciliation, and five assertions. A table can pass all of it and still be wrong, because every one of those checks tests a property you thought to check.

The review pass covers what you did not think of. It needs a name and a fixed order for the same reason a pre-flight checklist does: performed from memory, under time pressure, it silently becomes “the parts I remember.”

Call it the review pass. Six checks, always in this order, always by the person who will be accountable for the number.

Prompt first: turn the agent on its own output

Here is the transform you wrote and the output summary [paste].

Assume this table is WRONG and you have to find out how. Check
specifically for: statistics computed on a sample and reported as
whole-table; implicit type coercion on join keys; predicates present
in the SQL but absent from my written population rule; rows dropped
by watermark or join semantics; and aggregations that do not match
my definition's stated calculation.

Output a table: defect class, whether it is present, the evidence
query, and what result would confirm it. Do not reassure me.

“Do not reassure me” earns its place. Asked to review its own work, an agent’s default is a summary of what it did and why it was appropriate. The adversarial framing plus the enumerated classes produces evidence queries instead of a defense.

The five defect classes

Each of these survives the assertions in Lesson 4.2, because each produces a table that is internally consistent.

1. Sampled truth

The agent profiled 10,000 rows and reported the finding as a fact about 3.24 million. The number is real; its scope is not. Tell: any statistic that arrived suspiciously fast on a large table. Check: re-run the specific query with no LIMIT and compare.

2. Silent coercion

A join between circuit_id stored as text in one table and integer in another. The engine coerces, most rows match, and the ones with leading zeros or trailing whitespace silently do not. Tell: a join key whose types you never explicitly confirmed. Check: count the unmatched rows on both sides — a LEFT JOIN ... WHERE right IS NULL that should return zero.

3. The reasonable filter

The agent added WHERE status <> 'TEST' because it looked sensible, and never mentioned it. It probably is sensible. But it is now an undocumented population rule that disagrees with your written definition, and nobody knows it is there. Tell: predicates in the final SQL that do not appear in your definition. Check: read every WHERE clause against the POPULATION line from Lesson 1.3.

4. Dropped late rows

Lesson 3.2’s watermark bug, or a JOIN that quietly became INNER. Rows that should be present are absent, and absent rows are invisible in every check that examines the rows you have. Tell: row counts slightly under expectation, consistently. Check: count on the source side with the same filters and compare — the assertion in 4.2 catches large gaps, not a steady 0.4%.

5. Plausible aggregation

AVG where the definition says weighted mean. SUM over a fan-out. Averaging an average. The result is the right order of magnitude and the wrong number. Tell: any aggregate over a joined table. Check: compute one value by hand, for one circuit, one month.

The pass, in order

1. Read every WHERE clause against the written population rule. Anything unaccounted for is a finding.
2. Confirm join key types on both sides, explicitly.
3. Verify one value by hand. One circuit, one month, computed independently. This single step catches defect classes 1, 2, and 5 more often than any other.
4. Compare source-side counts with the same filters applied.
5. Check the known values from the acceptance test.
6. Ask what would have to be true for the number to be wrong in a way none of the above would catch — then check that.

Twenty minutes. Step 3 is the one people skip and the one that pays.

The exercise

Build the July table, then have someone plant three defects from the five classes above — or plant them yourself and come back in a week. Run the review pass and find them.

Two things surface reliably. The first is that step 3, computing one value by hand, catches more than the other five steps combined. The second is that a defect you planted yourself is nearly invisible a week later, which is the honest lesson about reviewing your own work under time pressure.

Stop and escalate when a defect class cannot be ruled out from where you sit — a second reviewer on the one value you cannot verify by hand costs twenty minutes; shipping around the doubt costs the credibility of the number.

Key takeaway

Assertions test what you thought to check; the review pass covers what you did not. Five defect classes survive everything automated — sampled truth, silent coercion, the reasonable undocumented filter, dropped rows, and plausible-but-wrong aggregation — because each yields a table that is internally consistent and looks right. Run six steps in a fixed order, by the person accountable for the number, and never skip computing one value by hand. Turn the agent on its own output with an adversarial frame and a list of classes, and tell it not to reassure you. Lesson 4.4 makes the result something you can regenerate on demand.

LEADERSHIP DECISION   require a named review pass by the number's owner
                      before any AI-assisted data product publishes
PRACTITIONER ACTION   run the six steps in order; compute one value by
                      hand every time; read every WHERE clause against
                      the written population rule
SUCCESS MEASURE       planted defects found in the exercise; zero
                      published products carrying a predicate absent
                      from their definition