Testing Where Bugs Live
Updated
IntermediateWhat you'll learn
~18 min- Choose a test shape from your failure data, not from a diagram
- Exploit cheap real-dependency integration tests - the shift that redrew the shapes
- Structure E2E as data-driven page objects, and recognize a rotten config on sight
The argument to stop having
Pyramid, trophy, honeycomb, crab — the industry has spent a decade arguing test-suite shapes. The mature 2026 position, visible in how the serious shops actually write: the shapes are cost heuristics, not commandments, and the question that picks yours is empirical: where do your bugs actually come from? A team whose incidents trace to service-boundary misunderstandings needs integration weight no unit-heavy pyramid provides; a team whose bugs are domain-logic edge cases needs the opposite. Argue from your incident history, not from a conference slide.
What genuinely did change the economics — and why the center of gravity drifted integration-heavy: real-dependency integration tests got cheap. Container-per-test tooling (Testcontainers-class) spins up a real Postgres, a real queue, a real cache inside the test run — so the classic pyramid rationale (“integration tests are slow and flaky, minimize them”) aged badly. A test against a real database in a disposable container catches the SQL-dialect bug the mocked repository never could, at a cost that used to buy you one brittle staging-environment test.
Switch your workspace agent to Plan mode on your incident and bug tracker export, and start with the failure-data exercise:
Classify our last 25 escaped production bugs by the cheapest test layer that could have caught each one (unit, real-dependency integration, E2E, or non-test config/infra). Use that distribution to propose our target test-suite shape.E2E: the framework shape that scales
For the E2E slice, the 2026 default is settled (Playwright, by a wide margin - the npm numbers aren’t close), so the interesting question is structure. The reference estate’s framework demonstrates the folder layout that keeps E2E maintainable, and its three moves compose:
data/ per-SCENARIO JSON: inputs, expected outcomes - no codepages/ one PAGE OBJECT per screen: selectors + actions live here, and NOWHERE elsespecs/ a small number of DRIVER specs that iterate the data files against the page objects - three drivers, dozens of scenarios, not one spec per caseconfig/ environment targeting - tests retarget dev/test/prod by CONFIG, zero code editsThe payoffs: a new scenario is a JSON file (authorable by a tester or an AI CLI without touching code); a UI change is one page-object edit (not forty spec edits); and environment portability is configuration. If the role-×-scenario matrices of the Zero Trust training came to mind — yes: this framework is exactly the machine those matrices execute on, and data-driven scenarios are how DENY cells become one JSON file each.
And the same estate ships the perfect cautionary exhibit — a config worth showing your team as a what-not-to-do reading exercise: an absurd global timeout (hiding every slow-page regression), headless: false (needlessly slow in CI), a single worker (throwing away parallelism), and — the subtle killer — typo’d option keys that the framework silently ignores, so settings the team believed active had never applied at all. The lesson generalizes: test configs are code; review them like code; and verify settings take effect rather than merely exist (set an obviously-wrong value and watch it bite — config’s verify-by-denial).
For the second half of your artifact, switch your workspace agent to Plan mode on the crew-scheduling application:
Plan a data-driven Playwright E2E scaffold for Meridian's crew-scheduling app (page objects, one driver spec, and scenario JSONs including a dispatcher DENY check). Audit the Playwright config line by line and prove every setting applies.Check that the plan or draft covers:
- Two page objects (
pages/), one driver spec (specs/), and four scenario JSON files (data/), including an explicitDENYscenario proving the dispatcher role cannot reach the admin screen - Environment-targeting configuration (
config/) that switches betweendevandtestwithout code edits - A line-by-line audit of the Playwright config that deletes unjustified overrides, catches silently ignored keys, and proves active settings by failing on a deliberate bad value
If the scaffold puts CSS selectors directly inside the driver spec or scenario JSONs, follow up with: Move all selectors and screen actions into the page objects so scenario files hold only inputs and expected outcomes.
Test generation is one of AI’s genuinely strong delivery skills in 2026 (the E2E tooling now ships planner/generator/healer agents) — but generation amplifies your SHAPE decision. Hand an AI your failure-data distribution and the framework above, and it fills layers appropriately; hand it nothing and it produces the classic anti-pattern at machine speed: hundreds of shallow unit tests over mocks, zero coverage where your bugs actually live. Decide the shape (human, from data), generate the volume (AI, into the structure), review the assertions (human — an AI-generated test that asserts nothing meaningful is coverage theater).
Meridian's billing team runs the failure-data exercise: of 25 escaped bugs, 14 were service-boundary issues (billing ↔ meter-data API contract drift), 6 were SQL edge cases, 3 were UI journeys, 2 were config. Their current suite: 900 unit tests (heavy mocking), 12 E2E, no integration layer. What does the evidence prescribe?
Key takeaway
Pick the shape from the incident data, exploit the fact that real-dependency integration tests are now cheap, structure E2E as data + page objects + drivers so scenarios are files rather than code, and review test configs with the suspicion you’d give production code — including proving settings apply. Then let AI fill the shape you chose. Next: what to do when the suite you built starts lying.