Book Appointment

One Week on a Made-Up Ops Team: A KubernetesOps Story in Five Days

Healthcare Software

None of this team is real, and the hospital group is imaginary too; think of the cast as a composite sketch assembled to show how routine cluster care plays out when people try to do it properly. The thread running through it is KubernetesOps, the day-to-day upkeep of a Kubernetes cluster, and the characters are an infrastructure lead, two engineers and an operations manager who runs a network of orthopedic outpatient centers. We will keep the focus on IT process and leave anything clinical out.

Monday: the surprise in the inventory

The operations manager asked a short question: how far behind are we? The infrastructure lead did not know, so the team spent Monday building a list of what ran where. The cluster’s version, its add-ons, the date its current release would leave support, and the workloads that talked to retired interfaces. By evening the picture was uncomfortable but useful. The cluster trailed the newest release by three minors.

The xOps School Ops Atlas, summarized on its KubernetesOps entry, would call that a lag target missed: the healthy goal it gives is one version behind or fewer. It would also say what the lead already suspected, namely that a gap of three means three sequential hops, because the main API server cannot be skipped past a minor.

Tuesday: finding the retired calls

An engineer went looking for traffic aimed at deprecated interfaces. The Atlas describes two sources of evidence. One is a counter, exposed by the API server, of requests for deprecated interfaces; the engineer noted that it starts over whenever the server restarts, so an empty reading right after a restart means little. The other is a set of audit annotations that name the caller. The entry says beta interfaces stay available for a window of nine months or three minors following deprecation, and that window sets the deadline for any fix.

The search found two scheduling jobs and one reporting tool using retired calls. None was urgent, but all three had owners to chase. That was the day the group stopped treating upgrades as an infrastructure-only matter, and started treating them as a shared calendar.

Wednesday: the capacity argument

Finance had been asking why the cluster cost so much when graphs showed machines mostly idle. The second engineer pulled the numbers: requested capacity was high, actual use was low. The workloads had been given generous requests years ago, and because the scheduler places by requests, not actual consumption, the cluster could not pack them tighter. The Atlas treats those declared figures as binding promises, which explains why the packing problem could not be solved with a bigger budget alone.

The fix chosen was gentle. They switched on the vertical pod autoscaler with its mode set to Off, so it produced suggestions and altered nothing. The Atlas lists that as its approach to rightsizing, and adds that in-place resizing of running pods became stable in version 1.35, with two limits: the pod keeps its original quality-of-service tier, and giving memory back is attempted on a best-effort basis. The team also remembered a warning in the entry: the VPA project advises against pairing it with the horizontal autoscaler on the same CPU or memory metric. They chose to leave the horizontal one alone for now.

Readers who enjoy seeing how automation gets applied to fleets of working equipment will find a parallel in our piece on how engineers scale and manage fleets of working robots: the discipline is measuring first, adjusting second.

Thursday: the restore nobody had tried

The manager asked a question that made the room go quiet. If the cluster disappeared tonight, how long until the centers could book appointments again? The honest answer was that nobody knew. Backups of the cluster’s etcd data existed, and nobody had ever restored one. The lead wrote the question on the whiteboard and left it there all week, a small embarrassment that kept everyone honest, while the manager asked for the answer in minutes rather than adjectives.

The lead recalled the public outage the Atlas cites from March 2023, where an upgrade of a big, old cluster lost a label on its control-plane nodes, knocked out networking between machines and forced about five hours of recovery from backup. In that company’s own account, the restore procedure was not fully current and had not been used on a live system. Nobody wanted to be the second team to learn that on the day. So the team scheduled a drill for the following week, restoring into a scratch cluster and timing every step.

Friday: writing the plan down

By Friday the team had enough to build a plan using the thirty-sixty-ninety outline the Atlas suggests. Month one: finish the inventory and fix budgets that would block a machine drain. Month two: move all configuration into version control, carry out a first hop by following a written runbook and finish the restore drill. Month three: add an admission rule requiring requests, publish a small dashboard, apply the rightsizing recommendations and lay out a twelve-month upgrade calendar. The milestone was a production environment sitting within one version of current.

Other operations teams have written about similar tidying in a discussion of streamlining software and cloud operations, which is a decent sanity check against your own plan.

Three months later

The imagined follow-up is modest, which is the point. The first hop took longer than planned because one add-on needed a newer release. The second hop took half the time. The drill revealed that a step in the restore notes referred to a folder that no longer existed, and fixing the notes cost an afternoon instead of an outage. Requests were trimmed in stages, and the operations manager received one page each month with six numbers on it.

On the Atlas’s five-step maturity ladder, the team had moved from ad hoc to repeatable: configuration in version control and two consecutive hops done from a runbook. The next rung, managed, needs version lag at one or below, at least 98% of pods declaring requests and a drilled restore. They were close.

Takeaways for a real team

  • Count your lag before you plan; the number decides how many hops you owe.
  • Prove deprecated calls are gone with evidence, and remember the counter’s blind spot after restarts.
  • Treat requests as promises and rightsize with recommendations before automation takes the wheel.
  • Restore a backup on purpose, in daylight, with a stopwatch.
  • Give leadership a short monthly page rather than a long quarterly deck.