intellimetrics Learning
Modern DevSecOps Foundations Module 4 · Test and Release

Flakes, Shards, and Fast CI

Last reviewed · content updated

Intermediate

What you'll learn

~15 min
  • Detect flakes from run history and quarantine them with ownership, never deletion
  • Treat retries as telemetry - a passed-on-rerun is a data point, not a success
  • Make CI fast enough to gate: sharding, and selection by dependency graph

The lie that poisons everything downstream

A flaky test — pass/fail nondeterministically on the same code — is worse than a missing test: it trains the team to distrust red. Once “just re-run it” is the reflex, real failures ride the re-run wave into production, and every gate built in this training (branch policies, merge queues, canary analysis) inherits the noise. Flakes are also the number-one failure mode of merge queues specifically — a queue serializing merges through speculative CI turns every flake into a queue-wide stall. If Lesson 1.2’s rung 3 is in your plans, this lesson is its prerequisite.

The discipline, in three verbs

Detect — from history, mechanically. The definition is operational: same commit, both outcomes. Modern CI platforms increasingly surface this natively — Azure DevOps has built-in flaky-test detection and “passed on rerun” reporting; turn it on. No heroics needed: the run history already contains the truth; the tooling just has to read it.

ℹOn GitHub: no native flake detection

GitHub Actions ships nothing native for flake detection as of August 2026 — teams wire a third-party tracker or their test-reporting layer (several exist) to get the same same-commit detection and quarantine feed. The discipline is identical; only the plumbing is BYO.

Quarantine — with an owner and an SLA, never delete. The flaky test moves to a quarantine suite that runs but doesn’t gate: still executing (so its signal and its fix can be verified), no longer blocking merges (so it can’t teach red-distrust). Two non-negotiables make quarantine a repair bay rather than a landfill: every quarantined test gets an owner (the owning team, registry-style) and an SLA (fix or justify within N days — expiring, like every exception this training has taught). A quarantine list with no owners and no clock is deletion with better optics.

Meter — retries are telemetry. Bounded automatic retries (once, maybe twice) are a legitimate availability tool for the suite — if every retry is recorded and the passed-on-retry rate is a dashboard number someone owns. The distinction is the whole game: a retry that silently converts red to green hides the flake; a retry that logs itself measures it. Rising retry rate is your earliest warning that the suite is rotting — cheaper to read than the merge-queue stall that follows.

Fast enough to gate

Flake discipline keeps CI honest; speed keeps it bearable — a 40-minute gate gets bypassed culturally long before it’s bypassed technically. Two levers, in order of universality:

SHARD    split the suite across parallel jobs; merge the reports after.
         (Playwright: --shard + blob-report merge is the standard pattern;
         every CI platform parallelizes jobs - this is table stakes.)
SELECT   run only what the change can affect - and the mainstream 2026 form
         is GRAPH-BASED, not ML: monorepo tooling (Nx/Turbo/Bazel-class)
         computes affected projects from the dependency graph + remote
         cache. Honest scoping: it's a monorepo-tooling dividend; classic
         multi-repo estates mostly shard-everything (and the old ADO Test
         Impact Analysis is legacy-frozen - don't build on it). AI-based
         test selection exists as enterprise vendor tooling; treat vendor
         savings claims per the confidence rules of Lesson 1.1.

Sequence matters: stabilize before you accelerate. Sharding a flaky suite just parallelizes the lying — you reach the wrong answer faster. Quarantine first, then shard, then (if your repo shape allows) select.

The artifact - our flake policy, as config plus one page:
1. DETECTION on (platform-native where available), with the flake list
   auto-filed as work items to the owning team
2. QUARANTINE mechanics: the tag/suite split, runs-but-doesn't-gate wiring
   in the template library (a tier-1 step, naturally), owner + 14-day SLA
   fields required, expiry escalation
3. RETRY policy: max 1 automatic retry, every retry logged, passed-on-
   retry rate on the delivery dashboard with an alert threshold
4. SPEED: shard the E2E suite from 4.1 across 4 jobs with merged reporting;
   record the before/after wall-clock in the PR description
5. THE RULE, stated for humans: a red main is everyone's first priority;
   'it's probably the flaky one' is banned vocabulary once quarantine
   exists - if it's flaky it's quarantined, so red means REAL

That last line is the cultural payoff the machinery buys: quarantine’s deepest purpose is making the remaining red unambiguous.

KNOWLEDGE CHECK

Two weeks into the policy, the passed-on-retry dashboard shows the API suite's retry rate climbing from 2% to 9%, though the flake list is stable and quarantine is empty of API tests. A teammate shrugs: 'retries are handling it — that's what they're for.' What's the correct reading?

Key takeaway

Detect flakes from same-commit history, quarantine them with owners and expiring SLAs (runs-but-doesn’t-gate; never delete), and meter every retry so the passed-on-retry rate leads your quality dashboard. Stabilize, then shard, then select by graph where your repo shape allows. The prize is cultural: red means real. Next: shipping — with deploy and release finally uncoupled.

Search lessons