{ "axes": [ { "bad": "nobody has ever gone looking for the weak point", "good": "someone hostile did, deliberately, recently", "id": "adversarial-exposure", "locate": "Ask when someone last tried to break in on purpose, and what they were paid to find. If the answer is a scan nobody read, or a checklist somebody filled in, you are at the bad end. The distinguishing question is not whether you have security measures but whether anyone has adversarially tested them.", "order": 9, "title": "Adversarial exposure" }, { "bad": "one bad change takes the whole site down", "good": "one bad change reaches one cohort, briefly", "id": "blast-radius", "locate": "Look at your last three incidents and ask who was affected. If the answer was \"everyone\" all three times, you have no containment at all \u2014 which is worth knowing, because it means every change you ship is a full-population experiment whether you designed it as one or not.", "order": 3, "title": "Blast radius" }, { "bad": "six teams and a release train", "good": "one person ships one thing on a Tuesday", "id": "change-coupling", "locate": "Count the humans required for your smallest useful change to reach production \u2014 approvers, coordinators, and anyone who must merely be told. If that number is above two, the cost is not the meeting; it is that nobody makes small changes any more, so every change is large.", "order": 6, "title": "Change coupling" }, { "bad": "works on my machine", "good": "same input, same result, anywhere", "id": "determinism", "locate": "Run your suite twice on an unchanged tree, on a machine that is not yours. If you cannot do that in one command, you are at the bad end of this line \u2014 and of change coupling too, because you have no answer that does not route through a person.", "order": 4, "title": "Determinism" }, { "bad": "months, from a customer", "good": "seconds, from your editor", "id": "feedback-latency", "locate": "Do not ask how good your tests are. Ask when you last learned about a defect, and who told you. If the answer is a customer, a support ticket, or a dashboard you happened to be looking at, you are at the bad end of this line no matter what your coverage report says.", "order": 1, "title": "Feedback latency" }, { "bad": "one person knows, and they are on holiday", "good": "written down, and several people have done it", "id": "knowledge-distribution", "locate": "Pick your most critical subsystem and ask who could debug it at 3am. If it is one name, you do not have a documentation problem, you have a single point of failure that takes holidays. Now ask when that person last wrote anything down that someone else has since used.", "order": 7, "title": "Knowledge distribution" }, { "bad": "your users tell you", "good": "you knew before they did, and you know why", "id": "observability", "locate": "Think of the last incident, and ask how it started: with an alert, or with a human? Then ask a harder one \u2014 when the alert fired, could you tell which change caused it without guessing? Knowing something is wrong is the cheap half of this axis.", "order": 5, "title": "Observability" }, { "bad": "redeploy and pray, and hope the backup works", "good": "a flag flip, in seconds, with no deploy", "id": "reversibility", "locate": "Time yourself undoing the last change you shipped. Not describing how you would undo it \u2014 actually doing it, on a Tuesday, without waking anyone. If the answer involves a build, you are at the bad end; if it involves a database migration, you are further down than you think.", "order": 2, "title": "Reversibility" }, { "bad": "you find out from a regulator, a lawsuit, or a user who gave up", "good": "the obligation is written down and the path that satisfies it is tested", "id": "rights-and-compliance", "locate": "Ask what your software owes to someone who does not work for you \u2014 a user whose data you hold, a person using a screen reader, the holder of a licence you depend on. Then ask who checked, and when. If nobody can name the obligation, you are not at the bad end of this line, you are not on it.", "order": 10, "title": "Rights and compliance" }, { "bad": "fully booked, interrupt-driven, always behind", "good": "room in the week for the boring fix", "id": "slack", "locate": "Ask what happened to the last three things everyone agreed were worth fixing. If they are still on a list, you are at the bad end. Utilisation is the measurement to distrust here: a team at 100% is not efficient, it is a queue with no absorber, and queueing behaviour will do the rest.", "order": 8, "title": "Slack" } ], "controls": [ { "axes": [ "observability" ], "claim": "Paging on user-visible symptoms rather than internal causes reduces alert volume and raises the fraction of pages that correspond to a real problem, which is the number that determines whether anyone still reads them.\n", "cost": "medium", "decoy": "A comprehensive set of resource alerts. CPU, memory, disk, queue depth \u2014 each one a cause that sometimes produces a symptom, firing at 3am for a condition the system absorbed on its own.\n", "depth": 3, "evidence": [ { "shows": "Quantifies the detection gap in a real large service: \"\u224817% of incidents either lacked monitors or telemetry coverage\" and \"10% incidents were not detected due to bugs, e.g., high threshold, buggy feature, wrong configuration\". Correlating cause with detection: \"70% incident with code bugs does not have monitors\".\n", "src": "s-ghosh-fight-production-incidents" }, { "shows": "Names the distinction the control rests on \u2014 \"Your monitoring system should address two questions: what's broken, and why? The 'what's broken' indicates the symptom; the 'why' indicates a (possibly intermediate) cause\" \u2014 and the fatigue mechanism: \"When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page that's masked by the noise.\"\n", "src": "s-sre-book-monitoring" }, { "shows": "A success-ratio availability metric is measurably biased toward the most active clients \u2014 1% of users generate 62% of events \u2014 and that bias caused it to over- and under-state real outage severity compared with a per-user metric. What you count decides whether your alerting reflects what users experienced.\n", "src": "s-hauer-meaningful-availability" }, { "shows": "The inverse case: a code path shipped WITHOUT a flag sat globally live-but-dormant for about two weeks, and when an unrelated policy change activated it everywhere at once it crashed Service Control worldwide, taking down 50+ services. Google's own remediation was to make flags mandatory \u2014 to \"enforce all changes to critical binaries to be feature flag protected and disabled by default\".\n", "src": "s-google-cloud-2025-outage" } ], "field-notes": [ "A health check answered 200 while the service it fronted had come back with an empty configuration and every feature off. Everything the machine could feel was fine. Everything a user could feel was not.\n" ], "id": "c-alert-on-symptoms", "last-reviewed": "2026-08-19", "requires": [ "c-mark-every-deploy-in-telemetry" ], "status": "active", "tier": "II", "title": "Alert on what users feel, not on what machines feel", "would-settle-it": "Page volume and actionable-page fraction before and after a symptom-based rewrite of an alert set. The SRE literature asserts this strongly and reports it as practice rather than as a measured comparison.\n" }, { "axes": [ "rights-and-compliance", "feedback-latency" ], "claim": "Six mechanical failure categories account for 96% of automatically detectable accessibility errors and have done for seven consecutive years, so a scan wired into the change pipeline addresses the overwhelming bulk of what is detectable at very low cost.\n", "cost": "low", "decoy": "An annual accessibility audit. It produces a report against a snapshot of a site that has since changed, at a cadence that guarantees every regression ships and lives for up to a year.\n", "depth": 2, "evidence": [ { "shows": "95.9% of one million home pages had automatically detectable WCAG 2 conformance failures, averaging 56.1 errors per page. Six categories account for 96% of all detected errors and have done for seven consecutive years: low-contrast text (83.9% of pages), missing alternative text (53.1%), missing form labels (51%), empty links (46.3%), empty buttons (30.6%), missing document language (13.5%).\n", "src": "s-webaim-million" } ], "id": "c-automated-a11y-scan-in-ci", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "II", "title": "Scan for the six failures that account for almost everything", "would-settle-it": "Whether a CI-gated scan actually reduces the error rate of a site over time. The census below measures how common these failures are, not what removes them, and nobody appears to have measured a before-and-after on a population of sites that adopted a gate.\n" }, { "axes": [ "slack" ], "claim": "Reserving capacity for maintenance before commitments are made is the only form of slack that survives contact with a deadline, and a team at full utilisation cannot adopt any other control on this site.\n", "cost": "high", "decoy": "A rule that engineers may spend 20% of their time on maintenance. Unbooked time is not slack; it is the first thing that gets spent, and it is spent every week by whatever is most urgent.\n", "depth": 0, "evidence": [ { "shows": "\"people completed interrupted tasks in less time with no difference in quality\" \u2014 subjects compensated by working faster \u2014 but stress and frustration were significantly higher in both interruption conditions than baseline. The cost of interruption showed up in the human, not in the output.\n", "src": "s-mark-cost-of-interrupted-work" }, { "shows": "\"WIP correlates with lead time; that is, lower WIP indicates shorter lead times, which is consistent with claims in the literature\" \u2014 direct production-data support for the queueing mechanism the slack argument rests on.\n", "src": "s-sjoberg-wip-kanban" } ], "id": "c-budget-slack-for-the-boring-fix", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "II", "title": "Book the time for the boring fix before the week fills", "would-settle-it": "Defect and incident rates against reserved-versus-nominal maintenance capacity across comparable teams. Queueing theory predicts the shape strongly, but the prediction and the measurement are not the same thing and this site does not treat them as such.\n" }, { "axes": [ "blast-radius" ], "claim": "Exposing a change to a bounded population while watching a pre-agreed metric converts a full-population failure into a small one, but only where the metric and the abort threshold are chosen before the rollout starts.\n", "cost": "medium", "decoy": "A percentage rollout with nobody watching a metric. It has the mechanism and none of the control: the population is smaller, the exposure time is longer, and nothing is set up to notice.\n", "depth": 3, "evidence": [ { "shows": "Quantifies the detection gap in a real large service: \"\u224817% of incidents either lacked monitors or telemetry coverage\" and \"10% incidents were not detected due to bugs, e.g., high threshold, buggy feature, wrong configuration\". Correlating cause with detection: \"70% incident with code bugs does not have monitors\".\n", "src": "s-ghosh-fight-production-incidents" }, { "shows": "Correlating fault telemetry against in-flight staged rollouts caught 155 critical failures over eight months at 92.4% precision and 100% recall for data-plane rollouts, with the paper reporting that 99.2% of suspicious rollouts were blocked during canary or pilot stages rather than in production.\n", "src": "s-li-gandalf-safe-deployment" }, { "shows": "A concrete account of dark launch, flag-gated release and staged rollout operating together at scale: code live on all servers but hidden from users, a gating system deciding \"which users see which features of the code\" by criteria such as country or age, and rollout through internal-only, a few-thousand-machine stage, and full deployment \u2014 with rollback normally reverting a single commit.\n", "src": "s-facebook-dev-deploy-2013" }, { "shows": "The inverse case: a code path shipped WITHOUT a flag sat globally live-but-dormant for about two weeks, and when an unrelated policy change activated it everywhere at once it crashed Service Control worldwide, taking down 50+ services. Google's own remediation was to make flags mandatory \u2014 to \"enforce all changes to critical binaries to be feature flag protected and disabled by default\".\n", "src": "s-google-cloud-2025-outage" }, { "shows": "Configuration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.\n", "src": "s-facebook-config-management" }, { "shows": "An automated canary analysis system running at real production scale \u2014 roughly 200 judgments a day \u2014 gating deployments by statistically comparing time-series metrics between a canary and a baseline cluster rather than by eye.\n", "src": "s-netflix-kayenta-canary" } ], "id": "c-canary-a-cohort", "last-reviewed": "2026-08-19", "requires": [ "c-decouple-deploy-from-release", "c-mark-every-deploy-in-telemetry" ], "status": "active", "tier": "III", "title": "Turn it on for a cohort before everyone", "would-settle-it": "Distribution of users affected per incident, before and after adopting cohort rollout. The mean will not move much; the tail is the claim, and the tail is where these systems actually hurt.\n" }, { "axes": [ "blast-radius", "observability" ], "claim": "Fault injection produces evidence only when a randomly assigned control group receives the same traffic without the fault and a pre-declared metric is compared between the two; the design exists and is deployed, and its outcomes have not been published in quantified form by anyone.\n", "cost": "high", "decoy": "Chaos engineering as usually practised. Turn something off, watch dashboards, conclude the system is resilient. Without a control group there is nothing to compare the observation against, and the founding paper of the discipline describes the comparison design and then reports no results from running it \u2014 its one worked example is explicitly hypothetical.\n", "depth": 4, "evidence": [ { "shows": "A randomised design for fault injection in production: users are assigned to a treatment group routed to a fault-injected canary and a control group routed to an untouched baseline, with a declared metric compared statistically between them. Reported outcome is qualitative \u2014 several cases where timeouts were set incorrectly and fallbacks revealed a service to be more business-critical than its owner intended.\n", "src": "s-netflix-chap-chaos-automation" } ], "id": "c-chaos-experiment-with-a-control-group", "last-reviewed": "2026-08-19", "requires": [ "c-canary-a-cohort" ], "status": "contested", "tier": "IV", "title": "If you inject a fault, hold back a control group", "would-settle-it": "Any measured comparison at adoption level: incident rate or recovery time before and after a team adopted fault injection, or between teams that did and did not. A 96-source review of the literature concludes the field is dominated by unvalidated solution proposals with limited empirical research, and this entry stays contested until someone publishes a number.\n" }, { "axes": [ "rights-and-compliance" ], "claim": "Whether a consent dialogue is lawful is mechanically checkable \u2014 a visible reject option at the same level as accept, no pre-ticked non-essential boxes, no forced choice \u2014 and the overwhelming majority of deployed dialogues fail that check.\n", "cost": "low", "decoy": "A consent management platform. Buying one is the step everyone takes and it is not the control: measured across 680 live sites using the five most popular platforms, only 11.8% of the resulting dialogues met the minimum legal requirements. The default template is the problem, not the fix.\n", "depth": 0, "evidence": [ { "shows": "Only 11.8% of 680 consent dialogues met minimal legal requirements \u2014 a clearly visible reject option, no pre-ticked boxes, no forced-choice framing. In the embedded experiment, removing the opt-out from the first layer raised recorded consent by 22-23 percentage points, and forcing per-purpose controls onto the first layer lowered it by 8-20 points.\n", "src": "s-nouwens-dark-patterns-gdpr" } ], "id": "c-consent-meets-the-legal-minimum", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "II", "title": "Test the consent dialogue against the law, not the vendor default" }, { "axes": [ "feedback-latency", "adversarial-exposure" ], "claim": "Coverage-guided fuzzing against a continuously built target finds crash-class defects quickly and cheaply once wired in \u2014 median five days to detection \u2014 but the count of bugs found overstates the count of vulnerabilities prevented by roughly two orders of magnitude.\n", "cost": "high", "decoy": "A bug count. It is the number every fuzzing programme reports and it is a poor proxy for the thing you care about \u2014 of 23,907 bugs found across 316 projects in the largest public corpus, 98 received a CVE. Under half a percent.\n", "depth": 3, "evidence": [ { "shows": "Every one of 32 surveyed evaluations had methodological problems. In the reproduction, a statistically significant increase in unique crashing inputs (p=10^-10) corresponded to no significant increase in real distinct bugs (p=0.066) \u2014 the standard deduplication proxy overstates the difference between fuzzers.\n", "src": "s-klees-evaluating-fuzz-testing" }, { "shows": "Detection and repair are fast \u2014 a median of five days to detect a regression and 5.3 days to fix. And the yield is mostly not security: of 23,907 bugs, 98 received a CVE. Flaky bugs, 13% of the corpus, are overwhelmingly never fixed at 86% unfixed against 12% for the rest.\n", "src": "s-oss-fuzz-empirical-study" } ], "id": "c-continuous-fuzzing", "last-reviewed": "2026-08-19", "requires": [ "c-run-tests-under-sanitizers" ], "status": "active", "tier": "II", "title": "Fuzz continuously, and read the bug counts sceptically" }, { "axes": [ "reversibility", "blast-radius" ], "claim": "Separating the act of shipping code from the act of turning it on makes the risky half of a change reversible in seconds without a deploy, and lets the two halves be judged by different people at different times.\n", "cost": "medium", "counterevidence": [ { "shows": "A flag value that had been repurposed was safe on the seven servers that received the new deployment and catastrophic on the eighth, which did not: it re-activated dead code last used in 2003. SMARS \"routed millions of orders into the market over a 45-minute period\" and Knight \"lost over $460 million from these unwanted positions\". Flag reuse plus an inconsistent deployment produced an effectively irreversible blast radius.\n", "src": "s-knight-capital-sec-order" }, { "shows": "Flags accumulate as a maintenance liability even at a well-resourced organisation with a dedicated removal campaign: toggle count grew from 263 to 1,040 across the releases studied, developers made over 5,044 commits introducing or refactoring toggles, and of tracked release toggles only 20% were actually removed while 17% lingered in the source as debt.\n", "src": "s-rahman-feature-toggles-chrome" }, { "shows": "Configuration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.\n", "src": "s-facebook-config-management" } ], "decoy": "A staging environment. It answers \"does this run\", which is rarely the question, and it answers it with traffic that does not resemble yours. It also costs about as much to operate as flags do to adopt.\n", "depth": 2, "evidence": [ { "shows": "Cluster analysis separates self-described delivery performance into distinct profiles, and continuous delivery capabilities \u2014 trunk-based development, automated testing, deployment automation, loosely coupled architecture \u2014 predict cluster membership in a PLS-SEM model.\n", "src": "s-dora-2019-state-of-devops" }, { "shows": "A concrete account of dark launch, flag-gated release and staged rollout operating together at scale: code live on all servers but hidden from users, a gating system deciding \"which users see which features of the code\" by criteria such as country or age, and rollout through internal-only, a few-thousand-machine stage, and full deployment \u2014 with rollback normally reverting a single commit.\n", "src": "s-facebook-dev-deploy-2013" }, { "shows": "The inverse case: a code path shipped WITHOUT a flag sat globally live-but-dormant for about two weeks, and when an unrelated policy change activated it everywhere at once it crashed Service Control worldwide, taking down 50+ services. Google's own remediation was to make flags mandatory \u2014 to \"enforce all changes to critical binaries to be feature flag protected and disabled by default\".\n", "src": "s-google-cloud-2025-outage" } ], "id": "c-decouple-deploy-from-release", "last-reviewed": "2026-08-19", "requires": [ "c-one-step-rollback" ], "status": "contested", "tier": "II", "title": "Deploy dark; release with a flag", "would-settle-it": "Change-failure rate and recovery time before and after adopting flag-gated release in the same organisation, with deploy frequency held constant. Widely believed, widely deployed, and surprisingly hard to find measured in isolation.\n" }, { "axes": [ "change-coupling", "knowledge-distribution" ], "claim": "Breaking changes overwhelmingly ship without any prior deprecation signal \u2014 of 251 API versions that introduced a breaking change, 87.3% had deprecated nothing in the previous version \u2014 and how the signal is delivered predicts whether clients act on it.\n", "cost": "low", "decoy": "A changelog entry. It is written at the moment of the break, read by people who already upgraded, and reaches nobody still calling the thing you removed.\n", "depth": 4, "evidence": [ { "shows": "Of 251 API versions that introduced a breaking change, 87.3% deprecated nothing in the previous version. Among the 219 APIs with any deprecation content, only 45% supplied replacements for every affected operation, and just 3 used a proactive channel \u2014 informing callers at call time via headers or error codes rather than relying on documentation.\n", "src": "s-yasmin-restful-api-deprecation" } ], "id": "c-deprecate-before-you-break", "last-reviewed": "2026-08-19", "requires": [ "c-expand-contract-migrations" ], "status": "active", "tier": "II", "title": "Signal the removal before you make it" }, { "axes": [ "reversibility" ], "claim": "Splitting a schema change into an additive step, a migration period where both shapes are readable, and a removal step keeps every intermediate state rollback-safe, which an in-place change never is.\n", "cost": "medium", "decoy": "A migration with a tested down-script. It is reversible in the test environment, where there is no traffic and no data written between the two runs. In production the down-script is a data-loss event with good intentions.\n", "depth": 3, "evidence": [ { "shows": "11.7% of client packages and 13.9% of their releases were hit by a manifesting breaking change during non-major dependency updates, and 44% of those arrived in minor and patch releases \u2014 the versions the contract says are safe. Clients recovered in only 39.1% of cases.\n", "src": "s-mujahid-npm-breaking-changes" }, { "shows": "83.4% of upgrades comply with semantic versioning, and compliance has increased over time. Critically for blast radius: \"most BCs affect code that is not used by any client, and... only 7.9% of all clients are affected by BCs\" \u2014 breaking changes are common in API surfaces and their realised reach is far smaller than raw counts suggest.\n", "src": "s-ochoa-breaking-bad-semver" } ], "id": "c-expand-contract-migrations", "last-reviewed": "2026-08-19", "requires": [ "c-decouple-deploy-from-release" ], "status": "active", "tier": "II", "title": "Migrate in expand and contract, never in place", "would-settle-it": "Rate of incidents caused by schema change, and their recovery times, in teams that mandate expand-contract versus those that permit in-place. Schema changes are well recorded, so this is one of the more tractable studies on this site.\n" }, { "axes": [ "feedback-latency" ], "claim": "Writing a test that fails for the reported reason before writing the fix proves the defect is understood and the test can detect it, which a test written afterwards proves neither of.\n", "cost": "low", "counterevidence": [ { "shows": "\"Sequencing, the order in which test and production code are written, had no influence\" on quality or productivity. What did predict outcomes was granularity and uniformity of cycles \u2014 suggesting the benefits attributed to test-first come from working in fine, steady steps rather than from the ordering itself.\n", "src": "s-fucci-tdd-dissection" }, { "shows": "Test-first students wrote more tests, and writing more tests predicted higher productivity. On quality \u2014 the benefit most often claimed \u2014 the result was null: \"there are no significant quality differences between the two groups\", with the test-last group's median quality numerically higher.\n", "src": "s-erdogmus-test-first" } ], "decoy": "A test written straight after the fix. It passes, it looks identical in the diff, and it has never once demonstrated that it can detect the bug it is named for.\n", "depth": 2, "field-notes": [ "A mutation test reported that no mutants survived. It had never checked that its patch applied, so it was reporting on an unmodified program. A negative result requires proof the experiment ran.\n" ], "id": "c-failing-test-first", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "contested", "tier": "IV", "title": "Reproduce the bug in a failing test before fixing it", "would-settle-it": "Regression-recurrence rates for defects fixed with a demonstrated-failing test versus a written-after test, in one codebase over a long enough window to catch repeats. The data exists in any mature issue tracker; nobody appears to have mined it for this.\n" }, { "axes": [ "change-coupling", "slack" ], "claim": "Attaching an expiry to a flag at creation time converts removal from an unowned chore into a scheduled decision, which is the difference between a flag system that shrinks and one that only grows.\n", "cost": "low", "decoy": "A quarterly flag cleanup. It is a campaign rather than a mechanism, it competes with feature work every time it runs, and the measured outcome of exactly such a campaign at a well-resourced organisation was that most of the flags it targeted were still there afterwards.\n", "depth": 3, "evidence": [ { "shows": "Flags accumulate as a maintenance liability even at a well-resourced organisation with a dedicated removal campaign: toggle count grew from 263 to 1,040 across the releases studied, developers made over 5,044 commits introducing or refactoring toggles, and of tracked release toggles only 20% were actually removed while 17% lingered in the source as debt.\n", "src": "s-rahman-feature-toggles-chrome" } ], "id": "c-flag-has-a-removal-deadline", "last-reviewed": "2026-08-19", "requires": [ "c-decouple-deploy-from-release" ], "status": "active", "tier": "III", "title": "Every flag gets a removal deadline when it is created", "would-settle-it": "Flag population over time in organisations with an enforced expiry versus those relying on periodic cleanup. The Chrome data below measures the second case at scale; nobody appears to have measured the first.\n" }, { "axes": [ "determinism" ], "claim": "A suite that starts and tears down its own dependencies produces the same verdict on any machine, which is the precondition for treating red as information rather than as weather.\n", "cost": "high", "decoy": "A shared staging database that everyone's tests point at. It makes the suite pass on a laptop, which is the thing people are trying to fix, and it couples every engineer's results to every other engineer's timing.\n", "depth": 1, "evidence": [ { "shows": "Bit-for-bit identical builds are achievable at scale: over 95% of 30,000+ packages, verified by deliberately varying about thirty environmental parameters between two builds and diffing the result. Names the concrete non-hermetic causes found this way \u2014 embedded timestamps, filesystem ordering, locale-dependent hashing, build paths.\n", "src": "s-lamb-reproducible-builds" } ], "field-notes": [ "A test asserting a fixture against a live service turned every upstream change into a failure in code that had not moved. The suite was correct about something; it was not correct about the thing it claimed to test.\n" ], "id": "c-hermetic-test-environment", "last-reviewed": "2026-08-19", "requires": [ "c-one-command-build", "c-pin-every-dependency" ], "status": "active", "tier": "II", "title": "Tests bring their own world", "would-settle-it": "Flake rate and mean time-to-diagnose for the same suite run against shared versus per-run infrastructure, in the same organisation, over a quarter. This is measurable today by anyone who has just migrated and kept their CI history.\n" }, { "axes": [ "determinism", "reversibility" ], "claim": "Non-idempotent infrastructure scripts are a defect category with no analogue in application code \u2014 practitioners surveyed rated it the most recognisable of eight categories \u2014 and they fail precisely when re-run during recovery.\n", "cost": "medium", "decoy": "A change that succeeded. Running it once and getting the result you wanted says nothing about the second run, and the second run is what happens during every recovery, every retry and every re-provision.\n", "depth": 1, "evidence": [ { "shows": "An eight-category defect taxonomy including one with no application-code analogue \u2014 idempotency, \"defects that lead to incorrect system provisioning when the same IaC script is executed multiple times\" \u2014 which surveyed practitioners recognised most strongly. The most frequent category overall was erroneous configuration data, not wrong logic.\n", "src": "s-rahman-iac-defect-taxonomy" } ], "id": "c-idempotent-infrastructure", "last-reviewed": "2026-08-19", "requires": [ "c-pin-every-dependency" ], "status": "active", "tier": "II", "title": "Infrastructure code must be safe to run twice" }, { "axes": [ "observability" ], "claim": "Annotating dashboards with deploy events collapses the most common incident question \u2014 \"what changed?\" \u2014 from an investigation into a glance, at a cost of roughly one afternoon.\n", "cost": "low", "decoy": "A deploy log in a different system. It contains the same timestamps and requires a human to correlate them by eye, at 3am, under pressure, which is when eyes are worst.\n", "depth": 2, "evidence": [ { "shows": "Correlating fault telemetry against in-flight staged rollouts caught 155 critical failures over eight months at 92.4% precision and 100% recall for data-plane rollouts, with the paper reporting that 99.2% of suspicious rollouts were blocked during canary or pilot stages rather than in production.\n", "src": "s-li-gandalf-safe-deployment" }, { "shows": "Deriving the same delivery metrics from version control, CI and telemetry rather than from a survey exposes wide variation between individual services that a team-level self-report hides: \"team performance has limited representational capabilities for individual microservices performance.\" Also catalogues concrete weaknesses of survey measurement \u2014 subjective responses, coarse Likert scales, recall error, and poor scalability.\n", "src": "s-ruegger-automated-dora-metrics" }, { "shows": "Configuration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.\n", "src": "s-facebook-config-management" }, { "shows": "An automated canary analysis system running at real production scale \u2014 roughly 200 judgments a day \u2014 gating deployments by statistically comparing time-series metrics between a canary and a baseline cluster rather than by eye.\n", "src": "s-netflix-kayenta-canary" } ], "id": "c-mark-every-deploy-in-telemetry", "last-reviewed": "2026-08-19", "requires": [ "c-one-step-rollback" ], "status": "active", "tier": "III", "title": "Every deploy leaves a mark in the telemetry", "would-settle-it": "Time-to-identify-cause for incidents with and without deploy annotations, from one organisation's incident write-ups. Any team with a year of postmortems and a recent adoption date could measure this on themselves.\n" }, { "axes": [ "adversarial-exposure", "determinism" ], "claim": "The share of vulnerabilities caused by memory unsafety tracks the language new code is written in, not the language the codebase is written in \u2014 one platform went from 76% of vulnerabilities being memory-safety issues in 2019 to 24% in 2024 while the bulk of its code remained unsafe.\n", "cost": "high", "decoy": "Rewriting the old code. It is the version of this everyone imagines, it is usually infeasible, and it is not what produced the measured result \u2014 the codebase that halved its vulnerability rate is still majority memory-unsafe.\n", "depth": 0, "evidence": [ { "shows": "Memory-safety issues fell from 76% of vulnerabilities in 2019 to 24% in 2024, with the annual count dropping from 223 to 85 over 2019-2022, while the codebase remained majority memory-unsafe. The change tracks the language new code is written in rather than any rewrite of existing code.\n", "src": "s-android-memory-safe-languages" }, { "shows": "Around 70% of high and critical severity security bugs are memory-unsafety problems, and about half of those are use-after-free.\n", "src": "s-chromium-memory-safety" }, { "shows": "About 70% of the vulnerabilities addressed through a security update each year continue to be memory-safety issues, across patch years 2006 to 2018. The leading root causes since 2016 are heap out-of-bounds access, use-after-free, type confusion and uninitialised use.\n", "src": "s-msrc-memory-safety-trends" } ], "id": "c-memory-safe-language-for-new-code", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "II", "title": "Write new code in a memory-safe language" }, { "axes": [ "determinism", "blast-radius" ], "claim": "For a small number of components where a subtle concurrency or fault-tolerance defect would be unrecoverable, writing a formal specification and exhaustively model-checking it finds design errors that review and testing demonstrably do not, at a training cost measured in weeks rather than months.\n", "cost": "high", "decoy": "More integration tests. They explore the state space by sampling it, and the defects this control targets are the ones whose shortest reproduction is dozens of steps deep \u2014 a region no sampling strategy will reach by accident.\n", "depth": 1, "evidence": [ { "shows": "Model checking found subtle concurrency and fault-tolerance defects in production distributed systems that had already passed design review, code review and testing \u2014 including a DynamoDB data-loss bug whose shortest counterexample was 35 steps. Engineers were reported productive in TLA+ within two to three weeks.\n", "src": "s-aws-tla-formal-methods" } ], "id": "c-model-check-the-hard-algorithm", "last-reviewed": "2026-08-19", "requires": [ "c-write-the-decision-down" ], "status": "active", "tier": "III", "title": "Model-check the algorithm you cannot test your way out of", "would-settle-it": "A comparison of production incident rates between comparable systems built with and without formal specification. The available evidence counts bugs found before shipping, which is a different and easier question.\n" }, { "axes": [ "observability" ], "claim": "Two-sample testing between live traffic and a reference distribution, run over a pre-trained classifier's outputs rather than raw features, detects distribution shift without labels and outperformed the alternatives tested, particularly when few samples are available.\n", "cost": "medium", "decoy": "An accuracy dashboard. It needs labels, and in production the labels arrive late or never \u2014 by the time accuracy visibly drops you have been serving the degraded model for however long the label delay is.\n", "depth": 3, "evidence": [ { "shows": "Two-sample testing using a pre-trained classifier for dimensionality reduction performed best; the popular domain-classifier approach performed badly below about 100 samples and only caught up with more. Applied to the canonical benchmark's own train/test split, the detector found a real shift traced to how one digit was drawn between collection batches, at p = 2.7e-10 \u2014 though its own malignancy test judged it harmless to accuracy.\n", "src": "s-rabanser-failing-loudly" } ], "id": "c-monitor-input-distribution-shift", "last-reviewed": "2026-08-19", "requires": [ "c-validate-data-at-the-boundary" ], "status": "active", "tier": "I", "title": "Watch the input distribution, and test whether the shift matters" }, { "axes": [ "feedback-latency", "determinism" ], "claim": "Seeding faults into the lines a change touches, and surfacing the survivors during review, measures whether a suite can actually detect a defect \u2014 something coverage percentage does not measure \u2014 at a cost that only becomes bearable when the analysis is scoped to the diff.\n", "cost": "high", "counterevidence": [ { "shows": "Asking a different statistical question gives a different answer: across projects, statement coverage is the best predictor of mutation kills (R\u00b2=0.94 for original suites). The two coverage papers are not in direct contradiction \u2014 they measure across projects versus within a project with size controlled.\n", "src": "s-gopinath-coverage-predicts-kills" } ], "decoy": "A code coverage target. It is the most widely deployed proxy for suite quality and the evidence for it is genuinely contested: once test-suite size is held constant, the correlation between coverage and fault detection falls from strong to somewhere between low and moderate. Coverage finds untested code well. It does not tell you whether the tests you have can detect a fault.\n", "depth": 2, "evidence": [ { "shows": "Each property-based test kills about 50 times as many mutants as the average unit test across 40 projects. Properties checking exceptions, collection inclusion or types were over 19 times more effective than other categories while making up only 10% of those written. 76% of all mutations caught were caught within the first 20 generated inputs.\n", "src": "s-ravi-property-based-testing-python" }, { "shows": "Once test-suite size is held constant the correlation between coverage and fault-detection effectiveness falls from moderate-to-high into the low-to-moderate range: \"it is not generally safe to assume that effectiveness is correlated with coverage\". Suite size is the confound doing much of the work.\n", "src": "s-inozemtseva-coverage-not-correlated" }, { "shows": "Scoping mutation analysis to the diff and suppressing uninteresting mutants is what made the technique deployable at scale, surfaced inside mandatory code review, with roughly 75% positive developer feedback on the mutants shown.\n", "src": "s-petrovic-state-of-mutation-google" }, { "shows": "Developers exposed to mutation testing wrote measurably more tests than a coverage-only control group, and the additional tests were more effective \u2014 mutant survivability declined with exposure. In a sample of 1,502 real high-priority production bugs, mutants coupled with roughly 70%.\n", "src": "s-petrovic-mutation-practices" }, { "shows": "Mutation score correlates with real-fault detection independently of code coverage, and 73% of the real faults studied were coupled to at least one generated mutant.\n", "src": "s-just-mutants-valid-substitute" } ], "id": "c-mutation-test-the-diff", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "I", "title": "Mutation-test the diff, not the repo", "would-settle-it": "Superseded by the evidence below.\n" }, { "axes": [ "knowledge-distribution", "change-coupling" ], "claim": "The number of low-ownership contributors to a component predicts its pre-release and post-release failures after controlling for size, complexity and churn \u2014 in two large systems it was more strongly correlated with defects than any other metric that organisation collected.\n", "cost": "low", "decoy": "Collective ownership of everything. It is a real and defensible position, and it is not the same thing as nobody having looked at a component in a year. The measurable risk is not shared ownership; it is the count of contributors with almost no stake in a component who each changed it once.\n", "depth": 0, "evidence": [ { "shows": "Adding ownership metrics \u2014 particularly the count of low-ownership 'minor' contributors \u2014 substantially increased explained variance in failures, from 26% to 46% pre-release in one system and from 24% to 70% in the other. Minor contributor count correlated with defects more strongly than any other metric the organisation collected.\n", "src": "s-bird-dont-touch-my-code" } ], "id": "c-name-an-owner-per-component", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "II", "title": "Give every component an owner, and watch the drive-by edits", "would-settle-it": "Superseded by the evidence below for organisations that already practise strong ownership. Whether the relationship holds where ownership is deliberately not stressed \u2014 much open source \u2014 is explicitly untested.\n" }, { "axes": [ "adversarial-exposure", "blast-radius" ], "claim": "Hard-coded credentials in configuration and infrastructure code are common, long-lived, and accepted as real defects by maintainers when pointed at \u2014 a mechanical check is the only thing that has been shown to find them.\n", "cost": "low", "decoy": "A policy saying not to. Prevalence was measured against codebases that all had such a policy: 21,201 smell occurrences across 15,232 infrastructure scripts, including 1,326 hard-coded passwords, with a median lifetime of 20 months and one surviving 98.\n", "depth": 0, "evidence": [ { "shows": "Committed secrets are pervasive and overwhelmingly go unrotated after exposure: more than 90% remained valid five days after leaking, and only 2.6% of leaks were revoked within an hour of notification \u2014 after 1.8 million pro-bono alert emails. Detection without enforced rotation is a weak control.\n", "src": "s-gitguardian-secrets-sprawl" }, { "shows": "21,201 smell occurrences including 1,326 hard-coded passwords. Of 1,000 sampled occurrences reported to maintainers, 104 drew a response and 67 were accepted as real defects. Median smell lifetime 20 months, with one hard-coded secret persisting 98.\n", "src": "s-rahman-iac-security-smells" } ], "id": "c-no-hardcoded-secrets", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "II", "title": "No secret is ever committed, and something checks" }, { "axes": [ "determinism", "feedback-latency" ], "claim": "A project where a clean checkout can be built and tested by one documented command has a shorter path to every other control on this site; one where it cannot will silently reintroduce environment drift into every later gate.\n", "cost": "low", "decoy": "A README with twelve numbered setup steps. It looks like documentation of a working process, and it is documentation of a process that only works for people who already know which steps are lying.\n", "depth": 0, "evidence": [ { "shows": "Before the migration, a new hire could clone the main repository, run setup and bootstrap scripts and have a working local instance in half a day. Mid-migration, cloning alone took 20 minutes and creating an environment upwards of 45. After optimisation, clone time fell to 90 seconds, environment creation to five minutes, and with prebuilds to 10 seconds.\n", "src": "s-github-codespaces-onboarding" } ], "field-notes": [ "A suite went green on a developer machine and red in CI for a week. The difference was a tool on the developer's `PATH` and not the runner's. Exit 127 \u2014 \"command not found\" \u2014 is non-zero, and non-zero satisfied about half the assertions the red-build alarm was making, so the alarm fired with the wrong reason attached and everyone debugged the wrong thing.\n", "A deploy that shipped a new binary but left the checkout's working tree alone made `git mv` a breaking change to a running service: the record naming the file updated instantly, the file itself never moved, and the service reported it missing. One fact, two clocks. The change contained no code.\n" ], "id": "c-one-command-build", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "III", "title": "One command builds and tests the project from a clean checkout", "would-settle-it": "The evidence below measures provisioning time, not the outcome anyone cares about \u2014 time to a new contributor's first merged change. Nobody appears to have measured that against a control.\n" }, { "axes": [ "reversibility" ], "claim": "A rollback that is a single command and has been executed deliberately in the last quarter converts a bad deploy from an incident into an inconvenience; one that exists only on paper does not.\n", "cost": "medium", "decoy": "A documented rollback procedure. Documents are not rehearsed, and the first rehearsal will be during an incident at 3am by whoever is on call, who has never read it.\n", "depth": 1, "evidence": [ { "shows": "A concrete account of dark launch, flag-gated release and staged rollout operating together at scale: code live on all servers but hidden from users, a gating system deciding \"which users see which features of the code\" by criteria such as country or age, and rollout through internal-only, a few-thousand-machine stage, and full deployment \u2014 with rollback normally reverting a single commit.\n", "src": "s-facebook-dev-deploy-2013" }, { "shows": "A flag value that had been repurposed was safe on the seven servers that received the new deployment and catastrophic on the eighth, which did not: it re-activated dead code last used in 2003. SMARS \"routed millions of orders into the market over a 45-minute period\" and Knight \"lost over $460 million from these unwanted positions\". Flag reuse plus an inconsistent deployment produced an effectively irreversible blast radius.\n", "src": "s-knight-capital-sec-order" }, { "shows": "Configuration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.\n", "src": "s-facebook-config-management" } ], "field-notes": [ "A service came back from a restart healthy and empty. It passed its health check, served every request successfully, and had turned every feature off. The rollback worked; the verification did not.\n" ], "id": "c-one-step-rollback", "last-reviewed": "2026-08-19", "requires": [ "c-one-command-build" ], "status": "active", "tier": "III", "title": "Rolling back is one step, and it is practised", "would-settle-it": "Mean time to recovery for organisations that rehearse rollback on a schedule versus those that document it, drawn from incident records rather than survey self-report. The DORA programme measures recovery time but does not isolate rehearsal as a variable.\n" }, { "axes": [ "rights-and-compliance", "observability" ], "claim": "An opt-out or deletion path is only real if it has been driven end to end by something that then checks the backing stores, because the front end confirms success independently of whether anything downstream changed.\n", "cost": "medium", "decoy": "Implementing the opt-out API. Adoption is not behaviour: of publishers who had implemented it, only 46.1% actually flipped their signal when a real opt-out was sent, and only 17.9% of downstream chains propagated it at all.\n", "depth": 1, "evidence": [ { "shows": "Participants routinely failed to find the correct path to opt-out and deletion controls and misunderstood what written requests required. Controls in account settings were markedly easier than those buried in privacy policies, and a substantial share fell back on help pages or emailing support to do something the site nominally offered as self-service.\n", "src": "s-habib-scavenger-hunt" }, { "shows": "Only 8.2% of publishers implemented the opt-out API at all, and only 3.6% stored the signal in the standard cookie. Where it was implemented, sending a global opt-out flipped the signal for only 380 of 825 publishers \u2014 46.1%. Across more than 1.2 million advertising chains, 17.9% propagated the opt-out downstream, and tracking-pixel deployment showed no statistically significant change when the signal was present.\n", "src": "s-aziz-wilson-ccpa-optout" } ], "id": "c-optout-and-deletion-are-tested-paths", "last-reviewed": "2026-08-19", "requires": [ "c-consent-meets-the-legal-minimum" ], "status": "active", "tier": "II", "title": "Drive your own deletion and opt-out paths with a canary account" }, { "axes": [ "determinism" ], "claim": "Pinning transitive dependencies to exact versions removes the largest source of builds that differ without the code differing, and makes a bisect a claim about your changes rather than about the internet that week.\n", "cost": "low", "decoy": "A lockfile that is generated but not committed. It produces identical builds on one machine and floating ones everywhere else, while looking from the outside exactly like the practice that would have worked.\n", "depth": 0, "evidence": [ { "shows": "Bit-for-bit identical builds are achievable at scale: over 95% of 30,000+ packages, verified by deliberately varying about thirty environmental parameters between two builds and diffing the result. Names the concrete non-hermetic causes found this way \u2014 embedded timestamps, filesystem ordering, locale-dependent hashing, build paths.\n", "src": "s-lamb-reproducible-builds" } ], "id": "c-pin-every-dependency", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "II", "title": "Pin every dependency to an exact version", "would-settle-it": "A comparison of build-reproducibility failure rates and mean bisect accuracy across projects with and without committed lockfiles, controlled for ecosystem. The ecosystems differ enough that a cross-language study would probably measure the package manager rather than the practice.\n" }, { "axes": [ "feedback-latency", "determinism" ], "claim": "A test that states a property and generates its own inputs detects substantially more seeded faults than an example-based test \u2014 around fifty times as many per test in the largest measured comparison \u2014 with the sharpest gains on properties that assert exceptions, membership or types.\n", "cost": "medium", "decoy": "More example tests. Each one encodes an input a person thought of, which is exactly the set of inputs the implementation was written against \u2014 the two are drawn from the same imagination and fail together.\n", "depth": 2, "evidence": [ { "shows": "Each property-based test kills about 50 times as many mutants as the average unit test across 40 projects. Properties checking exceptions, collection inclusion or types were over 19 times more effective than other categories while making up only 10% of those written. 76% of all mutations caught were caught within the first 20 generated inputs.\n", "src": "s-ravi-property-based-testing-python" } ], "id": "c-property-based-tests", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "I", "title": "State the invariant and let the machine pick the inputs" }, { "axes": [ "knowledge-distribution", "slack" ], "claim": "A team's shared belief that speaking up is safe predicts whether it discusses errors, asks questions and seeks feedback, which is the behaviour every other control on this site depends on to produce information.\n", "cost": "high", "decoy": "An anonymous feedback channel. It routes around the problem instead of changing it, and the information you most need \u2014 this deploy looks wrong, right now \u2014 is worthless anonymised and delayed.\n", "depth": 0, "evidence": [ { "shows": "Of the team-dynamics factors tested internally, psychological safety was the one most consistently associated with the four effectiveness measures used, ahead of composition variables like seniority, personality or team size.\n", "src": "s-google-project-aristotle" }, { "shows": "Team psychological safety is associated with learning behaviour \u2014 asking questions, discussing errors, seeking feedback \u2014 and that behaviour statistically mediates the relationship between safety and performance as rated by observers outside the team.\n", "src": "s-edmondson-psychological-safety" } ], "id": "c-psychological-safety", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "III", "title": "Make it safe to say the thing nobody wants to hear", "would-settle-it": "A study in a software engineering population, rather than an import from manufacturing plus a single unpublished corporate analysis. The construct is well established; its measured link to software outcomes specifically is not.\n" }, { "axes": [ "determinism", "feedback-latency" ], "claim": "Quarantining a flaky test with an expiry that fails the build converts an indefinite exclusion into a scheduled decision, which is the only mechanism that has been observed to make quarantine lists shrink.\n", "cost": "low", "decoy": "A skip annotation with a ticket number in the comment. The ticket ages, the annotation does not, and in two years nobody can say whether the test still describes the system.\n", "depth": 2, "evidence": [ { "shows": "Root-cause localisation for flaky tests reached 82% accuracy on 83 manually verified historical issues. Cites comparable flaky-failure rates elsewhere \u2014 4.56% at Google over 15 months, around 5% at Microsoft, 13% on TravisCI \u2014 suggesting a few percent of failures being flaky is a durable cross-organisation phenomenon rather than one company's artefact.\n", "src": "s-ziftci-deflake" }, { "shows": "At scale roughly 1.5% of test runs report a flaky result, almost 16% of tests show some flakiness at some point, and about 84% of the pass-to-fail transitions seen in CI are flakiness rather than real regressions.\n", "src": "s-micco-flaky-tests-google" }, { "shows": "Build failure is not a rare event: a mean failure ratio of 37% across 14 mature Java projects. Failures cluster \u2014 for 10 of the 14 projects, more than half and up to 80% of failed builds directly follow another failed build, and the fail ratio of the last 10 builds significantly predicts the next outcome in every project studied.\n", "src": "s-rausch-build-failures-in-ci" } ], "field-notes": [ "A test quarantined \"until the feature ships\" had never run. The suite used its recorded duration to decide what to parallelise, so its timing data \u2014 pure fiction \u2014 was shaping every other run.\n" ], "id": "c-quarantine-has-a-deadline", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "II", "title": "A quarantined test has a deadline, not a drawer", "would-settle-it": "Quarantine-list size over time in projects with and without an enforced expiry. This is trivially measurable from CI configuration history and would take an afternoon for anyone with access to a large monorepo.\n" }, { "axes": [ "knowledge-distribution", "observability" ], "claim": "The proportion of changes approved with no comments at all, and the proportion approved by their own author, predict post-release defects after controlling for size, complexity and churn \u2014 which review coverage by itself does not reliably do.\n", "cost": "low", "decoy": "A review coverage target. Getting every change reviewed is measurable, reportable, and \u2014 on its own \u2014 was not enough: components with high coverage were still defect-prone, and coverage explained defects in only two of four releases studied.\n", "depth": 2, "evidence": [ { "shows": "\"Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively.\" Changes approved with zero comments were significantly associated with defects across all three releases studied, as were self-approved and hastily reviewed changes.\n", "src": "s-mcintosh-review-coverage-participation" } ], "id": "c-require-discussion-not-just-approval", "last-reviewed": "2026-08-19", "requires": [ "c-second-pair-of-hands" ], "status": "active", "tier": "II", "title": "A review with no discussion is not a review", "would-settle-it": "Superseded by the evidence below. What remains open is whether requiring discussion produces discussion worth having, or merely produces comments.\n" }, { "axes": [ "reversibility", "observability" ], "claim": "A recovery path that has never been executed is a hypothesis, and the mechanisms most likely to fail are the ones that report success without ever having been asked to produce a usable restore.\n", "cost": "medium", "decoy": "Backups that report success. One organisation held five independent mechanisms \u2014 periodic dumps, volume snapshots, cloud disk snapshots, streaming replication, and a staging copy \u2014 and when it mattered, four had failed silently or were never enabled. The dumps had been broken for months and the failure emails were being rejected by the mail configuration.\n", "depth": 2, "evidence": [ { "shows": "Two recoveries of large user-data losses, both credited by the write-up to prior rehearsal. One restored 99%+ of data within hours of the initial estimate because similar situations had been simulated many times. The other recovered 436,223 of about 600,000 wrongly deleted audio references \u2014 161,000 were unrecoverable because they were deleted before ever being backed up \u2014 requiring 5,337 tapes recalled by truck, reaching 74% by day two, 99.95% by day three and seven days in total, with the tooling built and tested weeks earlier during a scheduled disaster exercise.\n", "src": "s-sre-book-data-integrity" }, { "shows": "Roughly 300 GB destroyed by a command run against the wrong host. Five redundancy mechanisms existed \u2014 periodic dumps to object storage, volume snapshots copied to staging, cloud disk snapshots, streaming replication, and a staging copy \u2014 and the write-up states the process of both finding and using backups failed completely. The dumps had been silently failing for months on a version mismatch, and their failure notifications were being rejected by mail policy. Recovery used a staging snapshot from six hours earlier, copied over a throttled link, taking about eighteen hours, with final loss estimated at 5,000 projects, 5,000 comments and about 700 users.\n", "src": "s-gitlab-2017-database-outage" } ], "id": "c-restore-drill-on-a-schedule", "last-reviewed": "2026-08-19", "requires": [ "c-one-step-rollback" ], "status": "active", "tier": "III", "title": "Restore from backup on a normal day, before you have to" }, { "axes": [ "adversarial-exposure", "feedback-latency" ], "claim": "Finding a committed credential does not remove the exposure \u2014 over 90% of leaked secrets were still valid five days after leaking \u2014 so revocation has to be automatic and time-bounded rather than a task assigned to whoever owns it.\n", "cost": "medium", "decoy": "An alert to the owner. It was measured at scale: 1.8 million notification emails, and 2.6% of leaks revoked within an hour. Detection that ends in a human being told is not a control, it is a statistic.\n", "depth": 1, "evidence": [ { "shows": "Committed secrets are pervasive and overwhelmingly go unrotated after exposure: more than 90% remained valid five days after leaking, and only 2.6% of leaks were revoked within an hour of notification \u2014 after 1.8 million pro-bono alert emails. Detection without enforced rotation is a weak control.\n", "src": "s-gitguardian-secrets-sprawl" } ], "id": "c-rotate-on-detection", "last-reviewed": "2026-08-19", "requires": [ "c-no-hardcoded-secrets" ], "status": "active", "tier": "II", "title": "A detected secret is revoked automatically, on a clock" }, { "axes": [ "determinism", "observability" ], "claim": "Running an existing test suite under address, thread and undefined-behaviour sanitizers surfaces a class of defect that neither the compiler nor the tests' own assertions can see, at a runtime cost low enough to run continuously.\n", "cost": "medium", "decoy": "Compiler warnings at maximum. They are free and worth having and they see the program as written, not as run \u2014 the defects here are use-after-free and data races, which exist only in an execution.\n", "depth": 2, "evidence": [ { "shows": "Over 300 previously unknown memory-safety bugs in ten months of continuous use, with a breakdown by class, at 73% average slowdown and 3.4x memory \u2014 low enough to run in unit tests and fuzzing, against roughly 20x for the previous generation of tools.\n", "src": "s-addresssanitizer" } ], "id": "c-run-tests-under-sanitizers", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "III", "title": "Run the suite under sanitizers, not just under the compiler" }, { "axes": [ "rights-and-compliance" ], "claim": "Licence incompatibility is predominantly a transitive property, so a check scoped to declared direct dependencies misses the majority of it \u2014 direct incompatibility rates ranged from 2.3% to 20.8% across seven registries while the indirect exposure sat underneath.\n", "cost": "low", "decoy": "Reviewing the licences of your direct dependencies. It is the review everyone actually does and it misses most of the exposure: 61.3% of license incompatibilities in one ecosystem-wide analysis came from transitive dependencies, and 6.62% of packages in another registry were violating the licence of an indirect dependency.\n", "depth": 1, "evidence": [ { "shows": "7.27% of package releases carried a licence incompatibility, and 61.3% of those were caused by transitive dependencies. When nine previously unknown incompatibilities were reported directly to maintainers, five were confirmed and fixed.\n", "src": "s-xu-pypi-license-incompatibilities" }, { "shows": "Direct incompatibility rates ranged from 2.3% in one registry to 20.8% in another. Copyleft exposure is overwhelmingly indirect: direct incompatibilities from one strong copyleft licence peaked at 0.04%, while 6.62% of packages in another registry were violating that licence via an indirect dependency.\n", "src": "s-pfeiffer-license-incompatibilities" } ], "id": "c-scan-transitive-licences", "last-reviewed": "2026-08-19", "requires": [ "c-pin-every-dependency" ], "status": "active", "tier": "II", "title": "Check licences through the whole graph, not the manifest" }, { "axes": [ "change-coupling", "blast-radius" ], "claim": "Enforcing a declared compatibility mode when a schema version is registered moves a producer-consumer break from the moment of consumption, distributed across every downstream reader, to the moment of change, localised to one author.\n", "cost": "low", "decoy": "Pairwise compatibility with the previous version. A chain of changes that are each compatible with their immediate predecessor can be jointly incompatible with a consumer two versions back \u2014 which is why transitive modes exist and why the non-transitive default is the one that bites.\n", "depth": 3, "evidence": [ { "shows": "Defines backward, forward and full compatibility modes, plus transitive variants that check against all prior versions rather than only the last, enforced when a schema version is registered \u2014 blocking a change that would break the declared contract with existing producers or consumers. Transitive modes exist specifically to prevent a chain of pairwise-compatible changes that are jointly incompatible.\n", "src": "s-schema-registry-compatibility" } ], "id": "c-schema-compatibility-at-the-topic", "last-reviewed": "2026-08-19", "requires": [ "c-validate-data-at-the-boundary" ], "status": "active", "tier": "IV", "title": "Reject an incompatible schema at registration, not at consumption", "would-settle-it": "How often incompatible-schema incidents actually occur, and what share registration-time enforcement prevents. The mechanism is documented only by its vendor and no adoption or effectiveness numbers appear to have been published by anyone.\n" }, { "axes": [ "rights-and-compliance", "knowledge-distribution" ], "claim": "A meaningful share of accessibility conformance is not mechanically decidable, so the critical paths have to be operated with assistive technology by a person, on a schedule, or the remainder is silently assumed to be fine.\n", "cost": "medium", "decoy": "A green automated scan. The engine's own maintainers put its ceiling at an average of 57% of issues \u2014 so a clean run certifies at most half the standard, and the half it cannot see is the half that needs judgement.\n", "depth": 3, "evidence": [ { "shows": "The tool's own documentation states: \"With axe-core, you can find on average 57% of WCAG issues automatically. Additionally, axe-core will return elements as 'incomplete' where axe-core could not be certain, and manual review is needed.\" By the same claim, an average of 43% of issues are not caught by automation.\n", "src": "s-axe-core-detection-ceiling" } ], "id": "c-screen-reader-tested-paths", "last-reviewed": "2026-08-19", "requires": [ "c-automated-a11y-scan-in-ci" ], "status": "active", "tier": "IV", "title": "Someone drives the critical path with a screen reader", "would-settle-it": "A measured comparison of barriers found by automated scanning against barriers found by assistive-technology testing on the same pages. The 57% figure is the tool vendor's own claim with no disclosed corpus or methodology, and this control rests on it until someone measures the split independently.\n" }, { "axes": [ "knowledge-distribution" ], "claim": "Having a second person perform a critical task under real conditions, at least once, converts a documented procedure into a demonstrated capability and surfaces the assumptions the document does not contain.\n", "cost": "medium", "decoy": "A runbook. It is written by the person who knows and read by nobody until an incident, at which point its unstated assumptions are discovered by someone with no context and a clock running.\n", "depth": 1, "evidence": [ { "shows": "\"recent departures from an organization were associated with increased probability of customer-reported defects\" after controlling for size, coupling, release count and experience. Notably, the number of newcomers joining was NOT a significant predictor \u2014 the cost is in people leaving, not in onboarding.\n", "src": "s-mockus-organizational-volatility" } ], "id": "c-second-pair-of-hands", "last-reviewed": "2026-08-19", "requires": [ "c-write-the-decision-down" ], "status": "active", "tier": "II", "title": "A second person has actually done the risky task", "would-settle-it": "Recovery time for incidents handled by the primary expert versus a second-trained person, and the rate at which runbooks are found wrong during use. Both are recorded in most incident processes already.\n" }, { "axes": [ "feedback-latency", "determinism" ], "claim": "Choose each test's scope by what it costs to run and what class of defect it can detect, rather than by conforming the suite to a fixed ratio; the ratios in circulation are targets that were published without a comparison against any alternative shape.\n", "cost": "medium", "decoy": "The test pyramid's ratio \u2014 70/20/10, or 80/15/5, or whichever numbers you were given. It is the most repeated prescription in test engineering and, as far as we have been able to establish, it has never been measured against outcomes. A ratio is not a reason.\n", "depth": 2, "evidence": [ { "shows": "Google states an internal aim of roughly 80% small, 15% medium and 5% large tests by test-case count, framed explicitly as its version of Mike Cohn's test pyramid.\n", "src": "s-google-swe-book-test-sizes" } ], "id": "c-shape-tests-by-cost", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "contested", "tier": "IV", "title": "Shape the suite by what each test costs and can catch", "would-settle-it": "A study measuring escaped-defect rate, suite runtime and maintenance cost across codebases with materially different suite compositions, controlling for size and domain. Searches for such a study \u2014 including for comparisons of the pyramid against the \"testing trophy\" or the ice-cream cone \u2014 returned nothing. This entry stays contested until someone measures a shape.\n" }, { "axes": [ "blast-radius", "slack" ], "claim": "Classifying requests by criticality and shedding the least important first converts a capacity incident from uniform failure into graded degradation \u2014 one reported outage held user-initiated availability above 99.4% while prefetch traffic was shed to 20%.\n", "cost": "medium", "decoy": "Autoscaling. It buys capacity on a timescale of minutes against an overload that arrives in seconds, and when the dependency you are overloading is the thing that is down, more instances of you does not help.\n", "depth": 5, "evidence": [ { "shows": "A concurrency limiter that categorises requests by criticality held user-initiated request availability above 99.4% during a real outage while shedding prefetch traffic to 20% availability.\n", "src": "s-netflix-load-shedding" } ], "id": "c-shed-load-by-priority", "last-reviewed": "2026-08-19", "requires": [ "c-tail-latency-slo" ], "status": "active", "tier": "IV", "title": "Decide in advance what to drop when you run out of capacity", "would-settle-it": "Whether graded shedding actually reduces user-visible impact across many incidents, rather than in the one that got written up. The available account is a single outage reported secondhand, with no measure of how often the limiter sheds traffic it should have served.\n" }, { "axes": [ "feedback-latency", "determinism" ], "claim": "A static type checker with null-strictness enabled detects a measurable minority of real defects before they ship \u2014 around 15% of public bugs in the best-controlled study \u2014 at an annotation cost measured in a couple of tokens per site.\n", "cost": "high", "counterevidence": [ { "shows": "After cleaning, only 4 of the original 11 languages retained a significant association with defects, and even then the association is described as exceedingly small. The widely repeated TypeScript result was largely a classification artefact: \"Out of 41 projects labeled as TypeScript, only 16 contained TypeScript\". An independent audit found a 36% false-positive rate in the original bug-fix commit labelling.\n", "src": "s-berger-language-quality-reproduction" }, { "shows": "TypeScript projects had significantly fewer code smells per line and lower cognitive complexity. But both bug-related hypotheses failed: no significant advantage in bug proneness or bug resolution time, with raw means running the other way \u2014 a bug-fix commit ratio of 0.126 for JavaScript against 0.206 for TypeScript.\n", "src": "s-bogner-js-vs-ts-quality" } ], "decoy": "Types without strict null checking. It is the migration everyone does first because it is the one that compiles, and it leaves out the single setting that accounted for the largest measured share of newly detectable defects.\n", "depth": 2, "evidence": [ { "shows": "Null-safety checking on every build at 1.15x build time, against 2.8x and 5.1x for sound alternatives, at about 12 annotations per thousand lines. After deployment null-pointer exceptions were under 5% of observed crashes, and of 100 production NPE traces over 30 days none were attributable to the checker's deliberate unsoundness on code it actually checked.\n", "src": "s-nullaway-uber" }, { "shows": "Both checkers flagged 60 of 400 real shipped JavaScript bugs when annotations were added back \u2014 15%, 95% CI [11.5%, 18.5%] \u2014 at a median cost of two tokens per bug. Strict null checking alone accounted for 22 additional detections, a 58% increase. Of what was missed, 55% were specification errors no type system could catch.\n", "src": "s-gao-to-type-or-not-to-type" } ], "id": "c-static-type-check-strict-null", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "contested", "tier": "I", "title": "Type-check on every change, with strict null checking on" }, { "axes": [ "feedback-latency", "determinism" ], "claim": "Detecting performance regressions requires change-point or confidence-interval methods over repeated runs rather than a threshold on a single measurement, and the precision-recall trade-off is a parameter to be chosen deliberately rather than a problem that gets solved.\n", "cost": "high", "decoy": "A threshold. \"Fail if p95 rises more than 5%\" is the obvious gate and it is the one that produces a 99.7% false-positive rate at scale, because run-to-run variance dwarfs the regressions worth catching.\n", "depth": 2, "evidence": [ { "shows": "Typical change-point detection would produce a 99.7% false-positive rate in that environment. 34% of naive subroutine-level flags are cost shifts \u2014 code moving between subroutines rather than getting more expensive. The median regression worth catching is 0.048% CPU, and regressions in that band collectively account for thousands of wasted servers a year.\n", "src": "s-fbdetect-meta" }, { "shows": "The production t-test-based gate runs at 12.5% false positives while missing about 6.8% of real regressions. Offline methods improve recall at a high cost to precision; the best ensembles improved F1 by 11% without removing the trade-off.\n", "src": "s-besbes-change-point-detection-mozilla" } ], "id": "c-statistical-perf-regression-gate", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "I", "title": "Gate performance statistically, and tune the false positives on purpose" }, { "axes": [ "rights-and-compliance", "observability" ], "claim": "A single accuracy figure conceals disparities large enough to make a system unusable for some people \u2014 audited against a balanced benchmark, three commercial classifiers had error rates up to 34.7% for darker-skinned women against 0.0% to 0.8% for lighter-skinned men.\n", "cost": "medium", "decoy": "Aggregate accuracy on a standard benchmark. The two benchmarks in common use when this was measured were 79.6% and 86.2% lighter-skinned subjects, so a model could post an excellent headline score precisely because the population it failed on was barely in the test set.\n", "depth": 0, "evidence": [ { "shows": "Error rates for darker-skinned women were 20.8%, 34.5% and 34.7% across the three systems, against a maximum lighter-skinned-male error of 0.8% and two systems at 0.0% and 0.3%. All classifiers performed better on males (8.1-20.6% error difference) and on lighter faces (11.8-19.2%). The two standard benchmarks these systems were likely evaluated against were 79.6% and 86.2% lighter-skinned, so the disparity was invisible by construction.\n", "src": "s-buolamwini-gender-shades" } ], "id": "c-subgroup-accuracy-audit", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "I", "title": "Report accuracy per subgroup, never as one number" }, { "axes": [ "feedback-latency", "knowledge-distribution" ], "claim": "Surfacing an analyser's finding inside the code review, against the lines it concerns, while the author still has the context, is what determines whether it gets acted on \u2014 and a false-positive rate above roughly a tenth destroys that regardless of how good the findings are.\n", "cost": "medium", "decoy": "A nightly analysis report with a dashboard. It finds the same defects and puts them somewhere nobody is standing, at a moment nobody is deciding anything. The measured version of this at Google was used by 35 developers in a year, 20 of them exactly once.\n", "depth": 2, "evidence": [ { "shows": "The false-positive rate is the lever that decides whether anyone acts on the output: \"False positives do matter. In our experience, more than 30% easily cause problems. People ignore the tool. True bugs get lost in the false. A vicious cycle starts where low trust causes complex bugs to be labeled false positives, leading to yet lower trust.\"\n", "src": "s-bessey-billion-lines-later" }, { "shows": "Where the analysis lands decides whether it is used. A prior non-integrated tool \"was used by only 35 developers in 2014 (and by 20 of those only once)\"; moving analysis into the review UI with a one-click not-useful button and a hard false-positive bar drove sustained engagement and a measurable decline in flagged patterns. Successful analysers held a not-useful rate between 0 and 3%.\n", "src": "s-sadowski-tricorder" } ], "id": "c-surface-analysis-at-review-time", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "III", "title": "Put static analysis in the review, not in a backlog", "would-settle-it": "Superseded by the evidence below, though the outcome measured there is developer response rather than escaped defects.\n" }, { "axes": [ "observability", "blast-radius" ], "claim": "In a fan-out architecture, per-component tail latency compounds into whole-request latency non-linearly, so an objective set on component averages can be met while the assembled request is far slower than any component.\n", "cost": "medium", "decoy": "A mean or median latency dashboard. It is the default panel in every monitoring tool and it is structurally blind to the effect that dominates user-visible latency in any fan-out system.\n", "depth": 4, "evidence": [ { "shows": "Fan-out amplifies tail latency non-linearly. A single leaf request's 99th percentile was 10ms; for 95% of leaf requests to finish, 70ms; for all of them, 140ms \u2014 so the slowest 5% accounted for half the total. With a 1% chance of a single server exceeding one second, a 100-way fan-out exceeds one second 63% of the time.\n", "src": "s-dean-tail-at-scale" } ], "id": "c-tail-latency-slo", "last-reviewed": "2026-08-19", "requires": [ "c-alert-on-symptoms" ], "status": "active", "tier": "III", "title": "Set the objective on the tail, measured where the request is assembled" }, { "axes": [ "feedback-latency" ], "claim": "Running the suite automatically on every proposed change, before review, shortens feedback latency from days to minutes and moves defect discovery off the reviewer, who is the most expensive and least reliable detector available.\n", "cost": "medium", "counterevidence": [ { "shows": "Selecting which tests to run, from failure and execution history, retained 70-80% of failure-detection while running a small fraction of the suite, and beat random selection at the same rate by roughly six times. Deferring some failing tests to post-submit added under 0.17% to post-submit load.\n", "src": "s-elbaum-regression-testing-in-ci" }, { "shows": "At Google's scale, testing every commit individually is infeasible: changes are batched into milestones cut roughly every 45 minutes, with observed delays up to 9 hours. Only 1.23% of test executions found a breakage or fix introduced by a developer, and of 5.5 million affected test targets only 63,000 ever failed.\n", "src": "s-memon-taming-google-scale-continuous-testing" } ], "decoy": "A nightly build. It runs the same tests and produces the same pass/fail, and it moves your feedback latency from minutes to a day while letting you say you have continuous integration. The batch is the problem, not the tests: one red nightly over twelve merged changes tells you the truth about none of them.\n", "depth": 1, "evidence": [ { "shows": "A learned selection model cut test executions by roughly a factor of three while \"guaranteeing that over 95% of individual test failures and over 99.9% of faulty changes are still reported back to developers\" \u2014 a concrete way to keep per-change feedback when running everything has become impossible.\n", "src": "s-machalica-predictive-test-selection" } ], "field-notes": [ "A gate that skipped silently \u2014 because the tool it invoked was absent \u2014 looked identical to a gate with nothing to catch. It reported green for weeks. A check that cannot distinguish \"nothing wrong\" from \"did not run\" is not a check; it is a light with no bulb.\n", "A test quarantined \"until the feature ships\" had never run once. Its timing data, which the suite used to decide what to parallelise, was fiction that nobody had reason to doubt because the number was there.\n" ], "id": "c-tests-run-on-every-change", "last-reviewed": "2026-08-19", "requires": [ "c-one-command-build" ], "status": "contested", "tier": "III", "title": "Every change runs the suite before a human looks at it", "would-settle-it": "A controlled comparison of defect escape rate and review throughput on comparable teams before and after introducing per-change automated runs. The DORA programme measures adjacent things at the organisational level, but the specific per-change claim is inferred from those results rather than tested by them, and this entry will stay at tier IV until something tests it directly.\n" }, { "axes": [ "adversarial-exposure", "knowledge-distribution" ], "claim": "Structured threat modelling taught to a team and then tracked produced mitigations that measurably blocked real attacks \u2014 541 unique intrusion attempts, five privileged account hijackings prevented, three public-facing vulnerabilities addressed over 120 days \u2014 with 20 of 25 participants still using it unprompted a month later.\n", "cost": "medium", "decoy": "A one-off workshop. The measurable question is not whether people can do it once with a facilitator in the room; it is whether any of them still do it thirty days later without being asked.\n", "depth": 1, "evidence": [ { "shows": "20 of 25 participants were still incorporating the method into daily duties 30 days after training without prompting. Over 120 days, participant-designed mitigations blocked 541 unique intrusion attempts, prevented the hijacking of five privileged accounts, and addressed three public-facing server vulnerabilities. Participants generated 147 distinct mitigation strategies, 64% of them new and unimplemented.\n", "src": "s-stevens-threat-modeling-enterprise" } ], "id": "c-threat-model-the-design", "last-reviewed": "2026-08-19", "requires": [ "c-write-the-decision-down" ], "status": "active", "tier": "II", "title": "Threat-model the design, and keep doing it after the training" }, { "axes": [ "adversarial-exposure", "slack" ], "claim": "Filtering vulnerability findings by whether the dependency actually ships, and by whether a version bump resolves it, separates the small urgent set from the large noisy one \u2014 81% were fixable by a simple version bump and only 1% sat on an unmaintained library needing a real mitigation.\n", "cost": "medium", "decoy": "The scanner's finding count. Matching advisory identifiers against a dependency tree overstates exposure: about 20% of dependencies flagged as vulnerable in one measured portfolio were test- or build-scoped and never deployed at all, so they could not be exploited in practice.\n", "depth": 3, "evidence": [ { "shows": "Separates presence from exposure: about 20% of dependencies flagged as affected by a known vulnerability are never deployed \u2014 test or build scope only \u2014 and cannot be exploited in practice. Of the deployed ones, 81% are remediable by a simple version bump, the library's own developers can fix 82% directly, and only 1% sit on an unmaintained library with no fix available.\n", "src": "s-pashchenko-counting-those-that-matter" } ], "id": "c-triage-by-reachability", "last-reviewed": "2026-08-19", "requires": [ "c-update-dependencies-on-a-bot" ], "status": "active", "tier": "III", "title": "Ask whether the vulnerable code ships before you treat it as urgent" }, { "axes": [ "knowledge-distribution" ], "claim": "Severe knowledge concentration is the normal state rather than a symptom of neglect \u2014 measured across 133 popular, actively maintained projects, roughly two thirds could be incapacitated by two people leaving \u2014 so it has to be measured deliberately rather than noticed.\n", "cost": "medium", "decoy": "A wiki page written by the one person who knows. It converts tacit knowledge into a document nobody has tried to use, which is a different and much weaker thing than a second person who has done the task.\n", "depth": 1, "evidence": [ { "shows": "\"recent departures from an organization were associated with increased probability of customer-reported defects\" after controlling for size, coupling, release count and experience. Notably, the number of newcomers joining was NOT a significant predictor \u2014 the cost is in people leaving, not in onboarding.\n", "src": "s-mockus-organizational-volatility" }, { "shows": "\"87 systems (65%) have TF \u2264 2\" \u2014 most popular, actively maintained open-source projects could be incapacitated by one or two departures. In 84% of valid survey answers respondents agreed the identified authors were the main authors of their system.\n", "src": "s-avelino-truck-factor" } ], "id": "c-truck-factor-floor", "last-reviewed": "2026-08-19", "requires": [ "c-name-an-owner-per-component" ], "status": "active", "tier": "II", "title": "Know the truck factor of anything you cannot afford to lose", "would-settle-it": "Whether a low truck factor actually predicts defects, delivery slowdown or abandonment. The measurement below establishes how common the condition is and explicitly does not test any downstream consequence.\n" }, { "axes": [ "change-coupling", "feedback-latency" ], "claim": "Keeping branches short forces changes to be small, and small changes are cheaper to review, cheaper to bisect, and cheaper to revert \u2014 the coupling cost is paid at branch length rather than at merge.\n", "cost": "medium", "decoy": "A well-run long-lived branch with regular merges from trunk. The merges keep it building; they do not make the change smaller, and size is the variable that matters.\n", "depth": 2, "evidence": [ { "shows": "Teams that merge to trunk at least daily, keep branch lifetimes under a day, and maintain fewer than three active branches report higher delivery performance, with the difference stated as statistically significant and replicated across two consecutive years: high performers' branch life and integration typically last hours, low performers' days.\n", "src": "s-puppet-dora-2017-state-of-devops" }, { "shows": "Cluster analysis separates self-described delivery performance into distinct profiles, and continuous delivery capabilities \u2014 trunk-based development, automated testing, deployment automation, loosely coupled architecture \u2014 predict cluster membership in a PLS-SEM model.\n", "src": "s-dora-2019-state-of-devops" } ], "id": "c-trunk-based-short-branches", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "II", "title": "Branches live hours, not weeks", "would-settle-it": "Defect escape rate against branch lifetime, controlled for change size, which is the confound: it may be that small changes are what helps and short branches are merely how you get them. Repository history makes this measurable at scale and the confound is what makes it interesting.\n" }, { "axes": [ "adversarial-exposure", "knowledge-distribution" ], "claim": "A single manual reviewer finds a small minority of known vulnerabilities \u2014 thirty reviewers on the same codebase averaged 2.33 of the confirmed set, about a fifth found none at all, and none found them all.\n", "cost": "medium", "decoy": "A senior reviewer. Experience did not predict accuracy in the measurement: more experienced reviewers were not reliably better at finding the planted vulnerabilities or at avoiding false positives.\n", "depth": 3, "evidence": [ { "shows": "No reviewer found all confirmed vulnerabilities. The average found was 2.33 with a standard deviation of 1.67, about 20% found none at all, and only 17% found the missing cross-site request forgery protection. False-positive rates were bimodal, and more experience did not reliably mean more accurate or effective.\n", "src": "s-edmundson-security-code-review" } ], "id": "c-two-reviewers-for-security-changes", "last-reviewed": "2026-08-19", "requires": [ "c-require-discussion-not-just-approval" ], "status": "active", "tier": "I", "title": "One reviewer is not enough for security-relevant code" }, { "axes": [ "adversarial-exposure", "feedback-latency" ], "claim": "Known-vulnerable dependencies sit in projects for a mean of 512 days before anyone acts, and automated update proposals collapse that to under a day \u2014 but only for the quarter that actually get merged, so the control is the bot plus a route to landing, not the bot alone.\n", "cost": "low", "decoy": "A security advisory feed. Awareness was measured and it is not the bottleneck \u2014 69% of developers whose projects carried a known-vulnerable dependency reported being unaware of it, and of those who did know, many deprioritised the upgrade as extra effort.\n", "depth": 2, "evidence": [ { "shows": "81.5% of systems keep their outdated dependencies. In the follow-up survey, 69% of interviewees said they were unaware of the vulnerability in their own dependency, and those who did know often deprioritised migration as extra effort and added responsibility.\n", "src": "s-kula-do-developers-update-dependencies" }, { "shows": "Known-vulnerable dependency versions sit undetected for a mean of 512 days, median 419. Bot-created security pull requests are merged only 25% of the time \u2014 44% are auto-closed, mostly superseded by a newer one, and 31% remain open \u2014 while manually created security pull requests are merged 74% of the time. Where merges happen they are fast: median under a day, and 78% of auto-merges complete in minutes.\n", "src": "s-rebatchi-dependabot-security-prs" } ], "id": "c-update-dependencies-on-a-bot", "last-reviewed": "2026-08-19", "requires": [ "c-pin-every-dependency", "c-tests-run-on-every-change" ], "status": "active", "tier": "II", "title": "A bot proposes the dependency update; a human still has to land it" }, { "axes": [ "observability", "blast-radius" ], "claim": "Validating incoming data against an inferred schema and a distribution baseline, before it reaches training or serving, converts a silent input-fitness failure into a visible one at the boundary where it entered.\n", "cost": "medium", "counterevidence": [ { "shows": "Average precision across tools was about 47% and average recall about 36%, with no tool dominant. Taking the union of all tools raised recall to 85-99% and severely degraded precision \u2014 99% recall at 39% precision on one dataset. A benefit-based ordering recovered 28% average precision for 3.5% recall loss. Some errors, such as a misspelled name or a wrong-but-plausible address, were structurally undetectable by any tool tested.\n", "src": "s-abedjan-detecting-data-errors" } ], "decoy": "Stacking every check you can think of. Coverage bought that way costs precision on a measured curve: across eight error-detection tools, taking the union of all of them reached 99% recall at 39% precision on one dataset. Most of what then fires is wrong, and an on-call who learns that stops reading.\n", "depth": 2, "evidence": [ { "shows": "Dimensionality reduction over multi-metric table health signals combined with exponential-smoothing forecasting, flagging one-step-ahead deviations across tens of thousands of tables where manual review is infeasible. Table-level alerts under this scheme are described as much rarer than per-metric alerts, framed explicitly as reducing alert fatigue.\n", "src": "s-uber-data-quality-monitor" }, { "shows": "Schema constraints plus inter-batch distribution distance, run ahead of training and serving. Over 30 days: new-column anomalies detected at about 100% with a 10% firing rate and 65% of fired alerts leading to a fix; missing features about 97% detected, 6% firing, 53% fixed; wrong data types about 98% detected, under 1% firing, 100% fixed. Over 70% of pipelines adopted schema-driven unit tests, which caught a 6% rate of code and schema mismatch.\n", "src": "s-tfx-data-validation" } ], "id": "c-validate-data-at-the-boundary", "last-reviewed": "2026-08-19", "requires": [ "c-tests-run-on-every-change" ], "status": "active", "tier": "II", "title": "Check the data is fit before you consume it" }, { "axes": [ "determinism", "blast-radius" ], "claim": "A documented consistency or safety guarantee is a claim, and generating histories and searching them for cycles finds violations of that claim in shipped systems at rates high enough to matter: 1,461 of 13,914 transactions in one measured run, under normal operation.\n", "cost": "high", "decoy": "The vendor's documentation. The system that lost roughly 10% of its transactions to isolation anomalies \u2014 with no faults injected at all \u2014 was advertised at the time as offering among the strongest data consistency, correctness and safety guarantees of any database available.\n", "depth": 2, "evidence": [ { "shows": "Where the previous-generation checker timed out or exhausted memory after a few hundred transactions, this approach checked histories of hundreds of thousands in tens of seconds \u2014 which is what makes the practice affordable. It revealed anomalies in every system tested.\n", "src": "s-elle-isolation-anomalies" }, { "shows": "Against a vendor claim of among the strongest data consistency, correctness and safety guarantees of any database available, the measured result was 1,461 of 13,914 transactions with cyclic dependencies \u2014 roughly 10% of transactions exhibiting anomalies during normal operation, without faults injected.\n", "src": "s-jepsen-mongodb-4-2-6" } ], "id": "c-verify-the-guarantee-adversarially", "last-reviewed": "2026-08-19", "requires": [ "c-hermetic-test-environment" ], "status": "active", "tier": "II", "title": "Test the guarantee your dependency advertises" }, { "axes": [ "knowledge-distribution" ], "claim": "A short record of what was decided, what was rejected, and why, prevents the most expensive class of rework: undoing a constraint whose reason was never written and is discovered by reintroducing the original problem.\n", "cost": "low", "decoy": "Documentation of how the system works. It describes the current state, which the code also does and does more accurately, and it says nothing about which constraints are load-bearing \u2014 so the next person removes one.\n", "depth": 0, "evidence": [ { "shows": "Argues that letting engineers give a full account \"without fear of punishment or retribution\" surfaces the real mechanisms behind a failure rather than stopping at human error, framed using Dekker's Just Culture and Hollnagel's treatment of human error as an effect rather than a cause.\n", "src": "s-allspaw-blameless-postmortems" }, { "shows": "Defines the practice: \"For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.\"\n", "src": "s-sre-book-postmortem-culture" } ], "id": "c-write-the-decision-down", "last-reviewed": "2026-08-19", "requires": [], "status": "active", "tier": "IV", "title": "Write down decisions, with the reasoning and the alternatives", "would-settle-it": "Frequency of decision-reversal-and-reversion in codebases with and without decision records. Hard to measure without reading a lot of history by hand, which is probably why nobody has.\n" } ] }