Two Clinics, Two AutomationOps Habits: A Comparison

Knee Care

Two physiotherapists can watch the same patient walk down the same corridor and describe two different walks. One sees a heel strike that lands a half-beat early; the other sees the compensation hiding six months up the chain. Automation estates reward the same kind of paired reading, which is why this piece compares two fictional clinics — call them Brookside and Ashgrove — whose scripting habits differ in ways that looked cosmetic right up until the week everything went wrong. The vocabulary for the comparison comes from AutomationOps as the Ops Atlas presents it: the working discipline for governing repeated machine actions as a production system, from selection through to retirement. Watch the two clinics walk, then read the gaits.

Brookside: AutomationOps as Folklore

Brookside’s IT story begins with Devraj, a systems administrator with a talent for shell scripting and a calendar nobody else can see. Over six years Devraj wrote forty-odd jobs: backup sweeps, an account-cleanup routine, the script that restarts the imaging queue when it sulks. Each lives in his home directory or a crontab he memorises. When Devraj took leave last winter, one job failed on day two and stayed failed for three weeks, because the only person who knew its path was walking the Camino. Access is simpler still: every job runs under one service login whose password sits in a spreadsheet shared with four people, two of whom left the company. Nothing about this is unusual, and that is precisely the finding worth sitting with. Brookside’s estate is not negligent; it is pre-disciplinary, the way walking is pre-gait-retraining. It works until load, absence or a bad Tuesday arrives.

Ashgrove: AutomationOps as an Estate

Ashgrove runs the same class of hospital on the same class of hardware, with one difference: three years ago its lead, comforted by nothing more than a bad quarter of incidents, began treating jobs like patients on a ward list. Every automated action lives in a registry the whole team can read, each line recording purpose, permitted targets, the identity it runs under, its danger grade and the date of its next review. Jobs execute through a shared runner, never from personal folders, so anyone on rotation can see state and history. The dangerous actions — the ones touching patient-adjacent data — sit behind an approval from a second engineer, and the estate’s halt mechanism is tested twice a year on a calendar entry signed by the head of IT, in the same spirit as a fire evacuation. Ashgrove did not buy a platform to do this; the registry is a versioned file and the runner is an off-the-shelf executor. What changed was habit, not software.

Where the Jobs Live

The contrast starts at the boringest possible layer — file paths — and that is exactly why it predicts everything else:

  • Brookside: jobs live where their author lives. Home directories, private crontabs, one laptop with a sticky note. Knowledge is personal, so absence is an outage.
  • Ashgrove: jobs live in the estate. One runner, one registry, review dates. Knowledge is institutional, so absence is a rota problem, not a failure.

The knee equivalent is unmistakable. A gait pattern that depends on one muscle group compensating forever will fail when that group tires; a pattern that shares load across the chain tolerates fatigue. Distributed knowledge is load-sharing for teams.

Failure Day, Side by Side

Consider how each clinic absorbs the same bad afternoon. A cleanup job begins deleting from the wrong directory tree — a filter typo, the classic kind. At Brookside, the job keeps running until someone notices missing files, because it has no notion of “too many errors”; the service login it uses can reach every system in the building, so its mistake can too. Reconstructing what happened afterwards takes days, because the only logs are whatever the script happened to echo. At Ashgrove, the same typo meets a different world: the job aborts after its third consecutive error and pages the on-call engineer; it works through targets in pods of ten behind a scout machine, so the damage is one pod, reviewed, before anything continues; and its registry line says precisely which directories it may touch. The forensic trail — who ran it, under what identity, with what approval — already exists. Recovery is an afternoon with a clear story in it, and stories you can retell are the only ones that teach.

Change Day Across a Fleet

The moment of maximum exposure in any estate is the fleet-wide change, so it makes the sharpest comparison. When a certificate authority rotates a root and every machine needs the new trust store, Brookside’s method is a loop and a prayer: run the command against all two hundred hosts in one motion, watch the output scroll, and hope the failures are legible. The loop has no memory of which hosts succeeded if the terminal closes, so verification is anecdotal. Ashgrove treats the same rotation as a procession — one scout machine first, inspected properly; then cohorts of ten with a health check at each boundary; a standing rule that three failures in any cohort halts the procession entirely; and the registry recording, host by host, what completed. The full rotation takes Ashgrove longer, and that sentence is where the whole comparison hides. It takes longer the way a correctly fitted brace takes longer than a stiff strap: the minutes are bought back, with interest, the first time something goes wrong underneath it.

Identity and Reach

Permissions are where automation fails loudest, so the comparison deserves its own moment. Brookside’s god login means every job is, effectively, every job: the account-cleanup routine could in principle touch payroll, and nothing in the estate would have objected. Ashgrove issues each job its own scoped identity — read here, write there, nothing anywhere else — so reach is a designed property rather than an accident of convenience. There is a public case that makes the stakes vivid. In 2021, a maintenance routine at one of the world’s largest social platforms was misled by a faulty audit check into believing the coast was clear, and it severed the company’s own network backbone. Every app the company owned went quiet across the planet for hours, and — the detail practitioners repeat quietly — the internal tools needed to fix the problem were down too, because they rode the same network. Reach without containment turned one bad check into a global outage. Ashgrove’s scoping would not have prevented that company’s bug, but it is the design that makes such blast radii structurally improbable: small identities, per-job halts, staged execution.

The Stop Story

Every estate has a stop story; the only question is whether it is a true story or a comforting one. Ask Brookside how a runaway job gets stopped and the answer is a shrug wearing confidence: Devraj would log in and kill it. Ask Ashgrove and the answer has a stopwatch attached — every job is haltable from the runner in one action, the whole estate has a master stop drilled twice yearly, and the on-call engineer holds standing authority to pull it without clearing it with anyone. That last detail matters more than it reads. In an emergency, permission chains are latency, and latency is damage. The Atlas puts braking drills on the same moral footing as fire drills: unpleasant to schedule, worthless until the day they are not.

What the Comparison Actually Argues

Notice what Brookside lacks is not talent, budget or effort — Devraj works harder than anyone at Ashgrove. What it lacks is structure, and structure is a choice. Clinics understand this instinctively, which is why they don’t assess a painful joint by vibes: a proper knee evaluation is systematic, graded and written down, and the write-up is what lets the next clinician continue the work. The same is true of ramping a load — nobody goes from walking to running on enthusiasm alone; they stage it, because staging is how humans survive repetition. Even fully robotic operations run on this principle, as the engineers who manage working robot fleets at scale will attest: registered machines, scoped permissions, staged updates, rehearsed stops. Brookside becomes Ashgrove in one deliberate quarter: a registry, a runner, scoped identities, grades, drilled halts. Two clinics, two gaits. Only one of them survives a marathon.

Leave a Reply