Flakes, Shards, and Fast CI
Updated
IntermediateWhat you'll learn
~15 min- Detect flakes from run history and quarantine them with ownership, never deletion
- Treat retries as telemetry - a passed-on-rerun is a data point, not a success
- Make CI fast enough to gate: sharding, and selection by dependency graph
The lie that poisons everything downstream
A flaky test — pass/fail nondeterministically on the same code — is worse than a missing test: it trains the team to distrust red. Once “just re-run it” is the reflex, real failures ride the re-run wave into production, and every gate built in this training (branch policies, merge queues, canary analysis) inherits the noise. Flakes are also the number-one failure mode of merge queues specifically — a queue serializing merges through speculative CI turns every flake into a queue-wide stall. If Lesson 1.2’s rung 3 is in your plans, this lesson is its prerequisite.
The discipline, in three verbs
Detect — from history, mechanically. The definition is operational: same commit, both outcomes. Modern CI platforms increasingly surface this natively — Azure DevOps has built-in flaky-test detection and “passed on rerun” reporting; turn it on. No heroics needed: the run history already contains the truth; the tooling just has to read it.
GitHub Actions had no documented native flake detection that we could find as of 2026-11-07 — teams wire a third-party tracker or their test-reporting layer (several exist) to get the same same-commit detection and quarantine feed. The discipline is identical; only the plumbing is BYO.
Quarantine — with an owner and an SLA, never delete. The flaky test moves to a quarantine suite that runs but doesn’t gate: still executing (so its signal and its fix can be verified), no longer blocking merges (so it can’t teach red-distrust). Two non-negotiables make quarantine a repair bay rather than a landfill: every quarantined test gets an owner (the owning team, registry-style) and an SLA (fix or justify within N days — expiring, like every exception this training has taught). A quarantine list with no owners and no clock is deletion with better optics.
Meter — retries are telemetry. Bounded automatic retries (once, maybe twice) are a legitimate availability tool for the suite — if every retry is recorded and the passed-on-retry rate is a dashboard number someone owns. The distinction is the whole game: a retry that silently converts red to green hides the flake; a retry that logs itself measures it. Rising retry rate is your earliest warning that the suite is rotting — cheaper to read than the merge-queue stall that follows.
Fast enough to gate
Flake discipline keeps CI honest; speed keeps it bearable - a 40-minute gate gets bypassed culturally long before it’s bypassed technically. Two levers in this acceleration structure, in order of universality:
SHARD split the suite across parallel jobs; merge the reports after. (Playwright: --shard + blob-report merge is the standard pattern; every CI platform parallelizes jobs - this is table stakes.)SELECT run only what the change can affect - and the mainstream 2026 form is GRAPH-BASED, not ML: monorepo tooling (Nx/Turbo/Bazel-class) computes affected projects from the dependency graph + remote cache. Honest scoping: it's a monorepo-tooling dividend; classic multi-repo estates mostly shard-everything (and the old ADO Test Impact Analysis is legacy-frozen - don't build on it). AI-based test selection exists as enterprise vendor tooling; treat vendor savings claims per the confidence rules of Lesson 1.1.Sequence matters: stabilize before you accelerate. Sharding a flaky suite just parallelizes the lying — you reach the wrong answer faster. Quarantine first, then shard, then (if your repo shape allows) select.
Switch your workspace agent to Plan mode on your pipeline templates and test configuration, and ask it to plan your flake policy and E2E sharding:
Plan our flake-management policy as pipeline configuration plus a one-page team guide: same-commit detection, non-gating quarantine with a 14-day SLA, metered retries, and 4-way E2E sharding.Check that the plan or draft covers:
- Same-commit flake detection enabled (or wired via test reporting) with newly detected flakes auto-filed to the owning team
- Quarantine suite wiring in the template library that runs without gating merges and requires an owning team plus a 14-day expiring SLA on every entry
- Bounded retry policy (maximum 1 automatic retry) where every retry logs to a
passed-on-retrydashboard metric with an alert threshold - Sharding the Lesson 4.1 E2E suite across 4 parallel jobs with merged blob reports and before/after wall-clock times recorded, plus the cultural rule that once quarantine exists, a red
mainis always treated as a real failure
If the retry configuration silently marks retried tests green without emitting a metric, follow up with: Log every passed-on-retry test run to our delivery dashboard and alert when the retry rate climbs.
That final cultural rule is the payoff the machinery buys: quarantine’s deepest purpose is making the remaining red unambiguous.
Two weeks into the policy, the passed-on-retry dashboard shows the API suite's retry rate climbing from 2% to 9%, though the flake list is stable and quarantine is empty of API tests. A teammate shrugs: 'retries are handling it — that's what they're for.' What's the correct reading?
Key takeaway
Detect flakes from same-commit history, quarantine them with owners and expiring SLAs (runs-but-doesn’t-gate; never delete), and meter every retry so the passed-on-retry rate leads your quality dashboard. Stabilize, then shard, then select by graph where your repo shape allows. The prize is cultural: red means real. Next: shipping — with deploy and release finally uncoupled.