A Twelve-Point KubernetesOps Audit for Front-Office Systems Teams

Health Tech

Clinics and hospitals rely on ordinary software to function: appointment calendars, billing screens, document stores, internal messaging. When that software runs on Kubernetes, someone must look after the cluster underneath it. This checklist is written for that administrative and technical layer only and says nothing about medical matters. It follows the structure of the KubernetesOps entry in the Ops Atlas, a reference that points out the name is informal and that the underlying skill set is usually called cluster administration. Each item below states what to check, what a pass looks like and what to do if it fails.

Section A: Versions and calendar

1. Inventory every cluster and its minor version. Pass: a single list, reviewed this month, with the running version and the date its support ends. Fail: anyone has to ask a colleague which version production runs. The Atlas notes that roughly fourteen months of patches accompany each release, and its lag target is one minor version or less.

2. Confirm the next upgrade has a booked date. Pass: a calendar entry exists for the next hop, and it is exactly one minor version ahead. Fail: the plan says “later this year”. The Atlas files that attitude among anti-patterns because each delay adds another hop that must be taken in sequence.

3. Verify the skew rules. Pass: no machine-level agent is newer than the API server, and none trails it by more than three minor releases. Fail: any agent that breaks that range. Fix by upgrading the laggard before touching the central components.

4. Search for deprecated interfaces. Pass: a report listing every manifest or chart that uses an interface announced for removal. The Atlas describes a metric that counts calls to such interfaces and audit annotations that tag them. Caveat: the metric only covers requests since the component last started, so run it after a long uptime, not right after a restart.

Section B: Capacity and requests

5. Count workloads with no requests. Pass: 98% or more of pods declare requests, the bar the Atlas sets for its third maturity level. Fail: a long tail of workloads with nothing declared, which the scheduler treats as free.

6. Compare requests with real use. Pass: you can state the usage-to-request ratio for each major workload and explain outliers. Fail: you rely on a feeling that the cluster is “full”. Use a recommender in observation mode to propose figures without acting on them.

7. Verify the autoscaling prerequisites. Pass: any workload scaled on processor use has a processor request, and nothing combines the horizontal and vertical approaches on the same metric. The Atlas also notes a five-minute stabilisation window before scale-down, which explains why replica counts shrink slowly after a peak.

Cost reviews fit here too. Finance teams already know how to plan recurring payments; a seasonal budget planning guide for festival spending gives a simple example of setting limits before the bills arrive, which is precisely what a capacity review does with cluster hours.

Section C: Disruption and resilience

8. List disruption budgets. Pass: none allows zero interruptions, and no two overlap on the same pod. Fail: either condition. The Atlas ties zero-interruption budgets to stalled drains, and overlapping budgets to server errors when a drain is attempted.

9. Review health probes. Pass: liveness checks inspect only the local process. Fail: a liveness check that calls a database or another service. The Atlas warns this pattern turns a minor outage into a cascade of restarts.

10. Check admission webhooks. Pass: none intercepts the system namespace or the networking component, and each failure policy has been chosen deliberately. By default, a webhook that cannot be reached blocks the request, so a badly placed one can freeze the cluster.

Section D: Recovery

11. Prove a restore. Pass: a backup of the cluster datastore has been brought back in a test environment within the last quarter, with the time recorded. Fail: backups exist but nobody has restored one. Among the Atlas’s cases is a 2023 incident at a social site that needed about five hours to recover, with its public account conceding the recovery notes were stale. Managed services do not remove this duty; they only change who takes the snapshot.

12. Check certificate expiry. Pass: each certificate’s expiry date appears on a shared calendar with an owner. On clusters built with the kubeadm tool, client certificates last one year, and a built-in command prints their remaining life.

How to run the audit

Allocate two people and a fixed time box. One reads the cluster; the other writes down the answers, because the person typing commands tends to skip documentation. Use read-only access wherever possible, since an audit should never change what it measures. Take screenshots or command output as evidence for each item and store them with the date. If a check cannot be answered, mark it as unknown, not as a pass. Unknowns often reveal more than failures do, since they point at places where nobody has looked.

Invite one person from outside the immediate team for the closing review. Fresh eyes ask about assumptions that the regulars no longer notice, such as why a particular add-on exists or who owns a certain namespace.

Scoring the audit

Give each item one point for a pass, half a point where the answer is partial and zero for a fail. Ten or more points suggests a team that has moved past ad hoc work. Seven or fewer says to start with sections A and D, since an unknown version and an untested restore carry the greatest risk. Run it again every quarter and file the results beside the configuration so that trends show up.

Turning failures into work

An audit without follow-up is theatre. For every failed item, record an owner, a due date and a verification step. Keep the list short: three open items at a time is more realistic than twelve. The Atlas’s ninety-day sketch is a good guide to sequencing, with inventory and the removal of drain blockers first, then configuration in version control and one practised upgrade hop, and finally dashboards and a twelve-month calendar.

The twelve checks take an afternoon the first time and under an hour thereafter. They will not make a cluster invulnerable, but they remove the commonest causes of unplanned downtime, and they give the people who depend on these systems a reasonable basis for trust.