intellimetrics Learning
Modern DevSecOps Foundations Module 6 · The Platform Team

Measure the System, Not the Vibes

Last reviewed · content updated

Intermediate

What you'll learn

~15 min
  • Wire the four keys plus rework to real pipeline events - on the tooling that actually exists
  • Read metrics through the amplifier lens: instruments for improvement, never weapons for ranking
  • Guard against the AI-era measurement traps the research already documented

Lesson 1.1 wrote a check; pay it

The opening lesson promised that measurement — not vibes — would referee Meridian’s delivery, then spent twenty lessons building things worth measuring. The bill arrives here, and the first fact is about plumbing: the four keys don’t come free. Neither of the big DevOps platforms ships a native four-keys dashboard (GitHub’s recent metrics investment has gone to AI-adoption reporting rather than four-keys; Azure DevOps has never had one) — so wiring is a deliberate act:

EVENTS    deployment markers are the atomic fact: every prod deploy from
          the Module 2 templates emits WHAT (artifact hash - 5.1's manifest),
          WHERE, WHEN, and WHICH COMMIT RANGE. (The OpenTelemetry CI/CD
          semantic conventions are the emerging standard shape for these -
          release-candidate stability; adopt expecting minor churn.)
STORE     the OSS default for this stack is a dedicated metrics collector
          (DevLake-class - it speaks Azure Pipelines natively); the
          observability vendors' DORA modules are the paid path; a
          warehouse query over your own deployment events is the honest floor
DERIVE    lead time (commit->deploy, per commit in the range, then
          aggregated - median or p75, say which), frequency (count),
          CFR (deploys that caused a failure - rollback, hotfix, OR a
          flag reversal, 4.3's primary un-release event), recovery
          (failure DETECTED -> restored, whether the gate tripped or a
          customer called), REWORK (DORA's definition: the share of
          deployments that were unplanned fixes for user-facing bugs -
          not PR churn, which is a different vendor metric)

Provenance makes the metrics trustworthy: because deploys carry artifact hashes and commit ranges (Module 5), lead time is computed from facts, not survey-estimated. Your delivery metrics inherit the integrity chain — that’s why this lesson lives after Module 5, not before.

Reading them: the amplifier lens

Numbers wired, now the discipline of reading them, which is where measurement programs die:

They’re instruments, not weapons. The moment a metric ranks teams or feeds a performance review, it stops measuring (targets get gamed — deploy-frequency theater is a real failure mode: batching trivial deploys to juice the count). DORA’s own evolution enforces this reading: the elite/high/medium/low ranks are retired in favor of team profiles — shapes to diagnose, not ladders to climb. The platform team reads its own product’s metrics the same way: adoption and delivery movement of teams ON the paved road versus off it is the platform’s ROI evidence (and the answer to 6.1’s “does any of this work?”).

The AI-era traps are documented; dodge them by name. Rework rate is the honesty metric — unplanned bug-fix deployments re-doing last sprint’s AI-generated feature aren’t throughput (industry PR-dataset research already shows measurable tech-debt and rework rises after AI adoption; vendor numbers, but directionally consistent with DORA’s stability finding). Perception is the other trap: the 19%-slower-feeling-20%-faster result means self-reported AI gains are the one instrument you never trust — which is precisely why the wiring above measures the system instead of surveying it. Supplementary frames exist (developer-experience metric suites that pair the four keys with experience dimensions); adopt them as diagnosis aids, same non-weapon rules.

The artifact - the measurement plan, one page + one pipeline:
1. the deployment-event emission step (tier-1 template, OTel-shaped
   attributes, artifact hash + commit range) added to the deploy stages
2. the collector wiring (or the warehouse-query floor), deriving the
   four keys + rework per SERVICE (not per person - ever)
3. the reading contract, written down: reviewed monthly by the team that
   owns the service, profiles-not-ranks, no individual attribution, no
   review-cycle use - signed by the platform lead so the contract
   outlives the enthusiasm
4. the platform's own scorecard: paved-road vs off-road delivery metrics,
   quarterly - the 6.1 feedback loop, closed with numbers
💬The floor is fine

If standing up a collector is this quarter’s bridge too far: a nightly query over deployment events in a spreadsheet-shaped table is a legitimate four-keys implementation, and teams have run serious improvement programs on exactly that. The discipline (events with provenance, service-level, profiles-not-ranks, monthly reading) is the program; the tooling is whatever makes the discipline cheapest today. Boring instrumentation honestly read beats a beautiful dashboard nobody trusts.

KNOWLEDGE CHECK

Meridian's CTO sees the new dashboard and asks for one addition: 'per-developer deployment frequency and lead time, so we can identify who's really benefiting from the AI tooling investment.' The measurement plan's reading contract has to answer. What does it say, and what's the counter-offer?

Key takeaway

Wire the four keys plus rework to provenance-carrying deployment events (the tooling won’t do it for you; the floor version is fine), then read them by contract: instruments not weapons, services not people, profiles not ranks, monthly by the owning team — with rework and stability as the honesty checks the AI era demands, and the paved-road-vs-off-road comparison as the platform’s own report card. One lesson left: the story of getting here from there.

Search lessons