What the Agent Got Wrong
Updated
AdvancedWhat you'll learn
~22 min- Run a named review pass over agent-authored work, in a fixed order
- Recognize the five defect classes that survive both testing and review
- Find planted defects in a table that passes every assertion
What it is: a fixed review pass, run by the person who owns the number, over work an AI assistant produced — with a defect exercise so the skill is practiced before it is needed.
What it buys: the specific failure mode of AI-assisted analysis — output that is fluent, confident, passes every automated check, and is wrong — gets caught by a human before it reaches a decision. This is the control that makes the speed safe.
What to fund: twenty minutes of review per data product, and the expectation that the reviewer is the person accountable for the number rather than whoever has time.
Why this needs a name
You have a definition, an acceptance test, a reconciliation, and five assertions. A table can pass all of it and still be wrong, because every one of those checks tests a property you thought to check.
The review pass covers what you did not think of. It needs a name and a fixed order for the same reason a pre-flight checklist does: performed from memory, under time pressure, it silently becomes “the parts I remember.”
Call it the review pass. Six checks, always in this order, always by the person who will be accountable for the number.
Prompt first: turn the agent on its own output
Switch your workspace agent to Plan mode on the transform file and output summary, and instruct it to audit its own work adversarially (if you do not have live exports or files handy, ask it to generate a small Meridian practice sample from the lesson’s scenario first):
Review the transform and output summary in this folder assuming the table is wrong. Check for silent defect classes that survive automated assertions, output an evidence query for each, and do not reassure me.Check that the plan or draft covers:
- Statistics computed on a sample and reported as whole-table truth
- Implicit type coercion on join keys that silently drops leading-zero or whitespace mismatches
- Reasonable-looking
WHEREpredicates present in the SQL but absent from your written population rule - Late rows dropped by watermark or
INNER JOINsemantics, and aggregations that diverge from your definition’s calculation - A structured table listing each defect class, whether it is present, the evidence query to test it, and the result that would confirm it
If the agent defends why its SQL choices were reasonable, follow up with: Skip the defense and output only the five defect classes, the detection SQL, and the confirming result.
“Do not reassure me” earns its place. Asked to review its own work, an agent’s default is a summary of what it did and why it was appropriate. The adversarial framing plus the enumerated classes produces evidence queries instead of a defense.
The five defect classes
Each of these survives the assertions in Lesson 4.2, because each produces a table that is internally consistent.
1. Sampled truth
The agent profiled 10,000 rows and reported the finding as a fact about 3.24 million. The number is real; its scope is not. Tell: any statistic that arrived suspiciously fast on a large table. Check: re-run the specific query with no LIMIT and compare.
2. Silent coercion
A join between circuit_id stored as text in one table and integer in another. The engine coerces, most rows match, and the ones with leading zeros or trailing whitespace silently do not. Tell: a join key whose types you never explicitly confirmed. Check: count the unmatched rows on both sides — a LEFT JOIN ... WHERE right IS NULL that should return zero.
3. The reasonable filter
The agent added WHERE status <> 'TEST' because it looked sensible, and never mentioned it. It probably is sensible. But it is now an undocumented population rule that disagrees with your written definition, and nobody knows it is there. Tell: predicates in the final SQL that do not appear in your definition. Check: read every WHERE clause against the POPULATION line from Lesson 1.3.
4. Dropped late rows
Lesson 3.2’s watermark bug, or a JOIN that quietly became INNER. Rows that should be present are absent, and absent rows are invisible in every check that examines the rows you have. Tell: row counts slightly under expectation, consistently. Check: count on the source side with the same filters and compare — the assertion in 4.2 catches large gaps, not a steady 0.4%.
5. Plausible aggregation
AVG where the definition says weighted mean. SUM over a fan-out. Averaging an average. The result is the right order of magnitude and the wrong number. Tell: any aggregate over a joined table. Check: compute one value by hand, for one circuit, one month.
Every one of these produces a table that is internally consistent, passes assertions, and looks right. None is a bug in the ordinary sense — the SQL is valid and the engine did what it was told. That is precisely why the review is a human pass rather than another test: the tests can only assert what somebody already thought to assert.
The pass, in order
- Read every
WHEREclause against the written population rule. Anything unaccounted for is a finding. - Confirm join key types on both sides, explicitly.
- Verify one value by hand. One circuit, one month, computed independently. This single step catches defect classes 1, 2, and 5 more often than any other.
- Compare source-side counts with the same filters applied.
- Check the known values from the acceptance test.
- Ask what would have to be true for the number to be wrong in a way none of the above would catch — then check that.
Twenty minutes. Step 3 is the one people skip and the one that pays.
The exercise
Build the July table, then have someone plant three defects from the five classes above — or plant them yourself and come back in a week. Run the review pass and find them.
Two things surface reliably. The first is that step 3, computing one value by hand, catches more than the other five steps combined. The second is that a defect you planted yourself is nearly invisible a week later, which is the honest lesson about reviewing your own work under time pressure.
Stop and escalate when a defect class cannot be ruled out from where you sit — a second reviewer on the one value you cannot verify by hand costs twenty minutes; shipping around the doubt costs the credibility of the number.
A table passes freshness, volume, uniqueness, referential, and range assertions, reconciles within tolerance, and matches both known values. What can you conclude?
Key takeaway
Assertions test what you thought to check; the review pass covers what you did not. Five defect classes survive everything automated — sampled truth, silent coercion, the reasonable undocumented filter, dropped rows, and plausible-but-wrong aggregation — because each yields a table that is internally consistent and looks right. Run six steps in a fixed order, by the person accountable for the number, and never skip computing one value by hand. Turn the agent on its own output with an adversarial frame and a list of classes, and tell it not to reassure you. Lesson 4.4 makes the result something you can regenerate on demand.
- LEADERSHIP DECISION
- require a named review pass by the number's owner before any AI-assisted data product publishes
- PRACTITIONER ACTION
- run the six steps in order; compute one value by hand every time; read every WHERE clause against the written population rule
- SUCCESS MEASURE
- planted defects found in the exercise; zero published products carrying a predicate absent from their definition