A knee clinic does not decide on surgery from a single glance at the joint. It follows a pathway: image first, grade what the images show, agree a plan, follow up on dated milestones, and change the plan only when the follow-up evidence says so. An automated test suite deserves the same clinical pathway treatment, and the discipline that supplies it is TestOps — the practice of operating your checks as a service whose users are the developers waiting on verdicts and whose promise is an answer that arrives quickly and tells the truth. The six steps below take a team from an unexamined suite to one it can defend in front of the business, and each step ends with evidence you can point at.
Step Zero: What TestOps Asks of You
Before the mechanics, one commitment. The people who write a change remain responsible for the checks that guard it; the operation standardises infrastructure and rules, never authorship. Delivery research keeps reaching the same conclusion from different angles — when a central team writes checks for everyone, pipelines spend more of their life broken and the codebase slowly becomes harder to test. With that settled, the steps are operational, not political.
Step One: Measure Before You Change Anything
Clinicians grade before they treat; engineers should too. Pull a month of CI history and compute five numbers:
- the share of red runs that were wrong — green on the second attempt, no fix in between;
- how often identical code produces different results;
- the wait for a verdict, both the typical case and the slow tail;
- how many checks are benched, and how long the eldest has been there;
- the spend — what one verdict costs once CI minutes are divided by verdicts produced.
Write these down. Every later claim of improvement leans on this baseline, and suites have a way of feeling slower or healthier than the data says they are.
Step Two: Give Every Stage a Budget
Map where feedback happens and price each stop: checks light enough for a laptop, measured in seconds; a pull-request stage held inside ten minutes, the ceiling delivery researchers keep rediscovering; a merge queue that verifies changes in landing order; the complete suite, split across machines, within half an hour after merge; an overnight sweep for whatever cannot run per change; and a few smoke probes once code is live. GitHub’s queue runs on the merge_group event — enable it, or the queue will disappoint you. Then enforce the rule that pays for itself: when a defect first shows up late, whatever fix you ship carries a faster check with it, placed earlier, so the same mistake cannot travel that deep again.
Step Three: Fix Environments and Data
Rank every dependency by how little fidelity still catches its failures. A genuine dependency used in-process tops the list; contrived fakes come next; then containers created for a single run and destroyed afterwards; then whole environments built per change; and finally the contract route, where a Pact can-i-deploy verdict settles whether two sides fit together without booting either one. Shared staging sits outside the ranking: humans click around in it, and nothing in a merge gate should depend on it. Data follows the same austerity. Each check makes what it needs through the product’s own interfaces at the start, marks it so no two runs or workers collide, and cleans up on the way out. Production data, where unavoidable, arrives trimmed and de-identified — never as a full copy.
Step Four: Handle Flakes in Public
Agree first on the test: one commit, two different results. That is the whole definition, and it converts shouting into measurement. The usual culprits are waits on things that show up late, state left behind by an earlier check, workers racing one another, checks that only pass in a certain order, clock boundaries, and selectors that match too much. Luo’s analysis puts the stakes plainly — only about one flaky check in five was born honest, which means most of the problem can be intercepted before a check ever gates anything, through a probation period of supervised runs. For checks already benched, attach three things: a name on the ticket, a date for the decision, and a hard ceiling on bench population. Fowler’s eight-checks-for-a-week remains the reference point, and GitLab takes the harder line, deleting benched checks after ninety days. Movement runs one direction — probation, gate, suspicion, bench, deletion — and never backwards without a fix.
Step Five: Store Results Where They Can Be Studied
The architecture has three layers: execution, signal and control. Execution is the machinery you built in steps two and three. Signal is the layer teams skip — a store of every attempt, keyed by the check’s stable name (class plus method), stamped with the commit and the attempt number, kept long enough to answer questions the CI log cannot. Without it, no flake rate, no bench automation, no history worth the name. Control is the gate and the bench registry, with one rule that saves careers: the gate must run even when other jobs fail. On GitHub Actions that means a condition such as if: !cancelled() on the gate job. Omit it, and branch protection will cheerfully merge a change whose checks never finished.
Step Six: Set a Trust Target and Prune
Write the promise down numerically: for example, ninety-eight verdicts in every hundred must be true across a four-week window — on twelve hundred runs, that tolerates twenty-four false alarms. A target turns vague unease into a budget with an owner. Add one column for escaped defects — of the bugs that reached users, how many slipped past the suite — so pruning never removes the wrong things. Then prune with conviction. A compact set of checks that everybody believes outweighs a sprawling set that everybody doubts; the delivery researchers have said as much for years, and every overloaded suite rediscovers it the expensive way.
Ninety Days of TestOps: What Good Looks Like
Slack’s mobile team supplies the endpoint worth aiming at: a main branch that sat green under a fifth of the time climbed to ninety-six percent once flake handling was taken seriously — same engineers, same product, different operation. Your own baseline from step one should show the same shape: fewer wrong reds, faster verdicts, a bench that empties. For the term itself and its boundaries, the TestOps entry in the Ops Atlas is the reference to cite. Continue from here with rethinking IT operations as one architecture, with why trusted information matters for patients — the parallel to verdicts people can believe is closer than it looks — and with the guide to DevOps transformation and training for the wider delivery context. Pathways beat heroic rescues in clinics and in CI alike; the six steps above are the pathway.