intellimetrics Learning
Modern DevSecOps Foundations Module 4 · Test and Release

Testing Where Bugs Live

Last reviewed · content updated

Intermediate

What you'll learn

~18 min
  • Choose a test shape from your failure data, not from a diagram
  • Exploit cheap real-dependency integration tests - the shift that redrew the shapes
  • Structure E2E as data-driven page objects, and recognize a rotten config on sight

The argument to stop having

Pyramid, trophy, honeycomb, crab — the industry has spent a decade arguing test-suite shapes. The mature 2026 position, visible in how the serious shops actually write: the shapes are cost heuristics, not commandments, and the question that picks yours is empirical: where do your bugs actually come from? A team whose incidents trace to service-boundary misunderstandings needs integration weight no unit-heavy pyramid provides; a team whose bugs are domain-logic edge cases needs the opposite. Argue from your incident history, not from a conference slide.

What genuinely did change the economics — and why the center of gravity drifted integration-heavy: real-dependency integration tests got cheap. Container-per-test tooling (Testcontainers-class) spins up a real Postgres, a real queue, a real cache inside the test run — so the classic pyramid rationale (“integration tests are slow and flaky, minimize them”) aged badly. A test against a real database in a disposable container catches the SQL-dialect bug the mocked repository never could, at a cost that used to buy you one brittle staging-environment test.

The failure-data exercise (your artifact's first half):
Pull the last 25 production incidents/escaped bugs. Classify each by the
CHEAPEST layer that could have caught it: unit (pure logic) / integration
(component + real dependency) / E2E (full journey) / none (config, infra -
Module 3's territory). The distribution IS your target test shape -
weighted by what each layer costs to run, a signal not a quota - and
the 'none' pile is a Module 3/5 work list, not a testing failure.

E2E: the framework shape that scales

For the E2E slice, the 2026 default is settled (Playwright, by a wide margin — the npm numbers aren’t close), so the interesting question is structure. The reference estate’s framework demonstrates the shape that keeps E2E maintainable, and its three moves compose:

data/            per-SCENARIO JSON: inputs, expected outcomes - no code
pages/           one PAGE OBJECT per screen: selectors + actions live here,
                 and NOWHERE else
specs/           a small number of DRIVER specs that iterate the data files
                 against the page objects - three drivers, dozens of
                 scenarios, not one spec per case
config/          environment targeting - tests retarget dev/test/prod by
                 CONFIG, zero code edits

The payoffs: a new scenario is a JSON file (authorable by a tester or an AI CLI without touching code); a UI change is one page-object edit (not forty spec edits); and environment portability is configuration. If the role-×-scenario matrices of the Zero Trust training came to mind — yes: this framework is exactly the machine those matrices execute on, and data-driven scenarios are how DENY cells become one JSON file each.

And the same estate ships the perfect cautionary exhibit — a config worth showing your team as a what-not-to-do reading exercise: an absurd global timeout (hiding every slow-page regression), headless: false (needlessly slow in CI), a single worker (throwing away parallelism), and — the subtle killer — typo’d option keys that the framework silently ignores, so settings the team believed active had never applied at all. The lesson generalizes: test configs are code; review them like code; and verify settings take effect rather than merely exist (set an obviously-wrong value and watch it bite — config’s verify-by-denial).

Your artifact, second half: scaffold the E2E framework for Meridian's crew-
scheduling app - two page objects, one driver spec, four scenario JSONs
(including one DENY scenario: the dispatcher role must NOT reach the admin
screen), environment config for dev/test. Then the config review: audit the
generated playwright config line by line - justify every setting, delete
what you can't justify, and prove one setting actually applies.
💡AI writes tests well - point it at the right layer

Test generation is one of AI’s genuinely strong delivery skills in 2026 (the E2E tooling now ships planner/generator/healer agents) — but generation amplifies your SHAPE decision. Hand an AI your failure-data distribution and the framework above, and it fills layers appropriately; hand it nothing and it produces the classic anti-pattern at machine speed: hundreds of shallow unit tests over mocks, zero coverage where your bugs actually live. Decide the shape (human, from data), generate the volume (AI, into the structure), review the assertions (human — an AI-generated test that asserts nothing meaningful is coverage theater).

KNOWLEDGE CHECK

Meridian's billing team runs the failure-data exercise: of 25 escaped bugs, 14 were service-boundary issues (billing ↔ meter-data API contract drift), 6 were SQL edge cases, 3 were UI journeys, 2 were config. Their current suite: 900 unit tests (heavy mocking), 12 E2E, no integration layer. What does the evidence prescribe?

Key takeaway

Pick the shape from the incident data, exploit the fact that real-dependency integration tests are now cheap, structure E2E as data + page objects + drivers so scenarios are files rather than code, and review test configs with the suspicion you’d give production code — including proving settings apply. Then let AI fill the shape you chose. Next: what to do when the suite you built starts lying.

Search lessons