A checklist can fail in two opposite ways. Either it runs so long that nobody completes it, or it is so loose that everyone scores full marks. This one aims for the middle. It is a DataOps self-audit for a hospital’s reporting and planning data, the kind of pipeline that feeds scheduling boards, supply orders and monthly summaries. It is deliberately non-clinical. Each section ends with a scoring note so you can see where to start.
The framework is lifted from the Ops Atlas page on DataOps, whose authors concede that the field’s vocabulary is unsettled and that the maturity ladder is their own construction. Treat the questions below as a structured conversation with your team, not as a certification.
Section A: Inventory and tiers
You cannot audit everything, so begin by deciding what counts. The Atlas splits outputs into three tiers: critical, important and exploratory. Work through these questions.
- Do we have a written list of the reports and tables that other teams act on?
- Is each one marked critical, important or exploratory, with a named owner?
- Does the critical group stay small enough that someone can recite it from memory?
- For each critical output, do we know which source systems feed it?
Scoring: four yes answers means you can move on. Any no means the rest of the audit is premature; spend your first month here.
Section B: Change control
This section covers what the Atlas calls the change loop, meaning edits to the code and queries that build your tables.
- Are pipeline definitions stored in a version-controlled repository, with history?
- Does a reviewer approve every edit before it goes live?
- Do automated tests run on each proposed edit, and do they run against sample or masked data, not live records?
- Is it possible to see which edit preceded a given bad number?
If your team uses dbt, the entry mentions a slim continuous-integration mode that tests only modified models and what depends on them, which keeps these runs fast enough that people do not skip them. A good primer on the broader technology stack is an article on hands-on skills for data-driven systems, which readers on the technical side may find useful context.
Scoring: if tests exist but run against production, count that as a no. The Atlas lists this as a risk in its own right.
Section C: The run loop
Here the audit turns to the data itself, which changes every time the pipeline executes with no release to review.
- For each critical output, are there checks for freshness, row volume and key columns that are never blank or duplicated?
- Do the checks block publication when they fail, rather than just logging?
- Is new data staged separately and promoted only after checks pass?
- When a source delivers late, does the pipeline reconcile partition counts to the source and re-read recent days?
- Can any load be run twice without creating duplicates?
- Is there a written agreement with each upstream team covering columns, meaning of values, timing and a contact?
Scoring: the Atlas ties its third maturity level to a concrete test, namely that no critical output was published unaudited for a whole quarter. Use that as your bar for this section. Four of six yes answers on critical outputs is a reasonable passing mark on a first pass.
A short aside on late-arriving data, since Section C asks about it. Streaming systems handle lateness with event-time watermarks, while batch pipelines usually use lookback windows that re-read the most recent days on each run; the Atlas cites a three-day incremental window in dbt as an illustration. Whichever you use, the audit question is whether someone chose the window deliberately and wrote it down.
Section D: Watching and reacting
- Do monitors that sit outside the scheduler confirm that critical tables arrived on time?
- Are alert thresholds built per weekday, or per season, where the data has a rhythm?
- Do we keep an incident log recording who found each fault and how long it ran undetected?
- Do we know the share of incidents that readers discovered before we did?
- Is there a dashboard showing arrival time and volume per source?
For the dashboard question, a walkthrough of building monitoring views in Grafana describes the practical side, and the same principles apply to whichever tool you already have.
Scoring: the incident log is the most valuable item here. Without it you cannot compute the detection figure, and the Atlas proposes it as the single most telling number. Its own downtime example showed three quarters of lost hours going to not knowing, so this section carries the heaviest weighting in the audit.
Section E: Safety and cost
The last section is the one most often skipped because it feels like housekeeping. It is also where reputational harm comes from, so give it the same attention as the rest.
- Do test and sandbox areas hold masked or synthetic records only?
- Do failing-row outputs go to a restricted place?
- Are expensive checks scoped to new partitions with wide sweeps scheduled less often?
- Do we track the cost of running each critical load?
Scoring: any unmasked personal data in a sandbox is an automatic red flag, whatever else scores well.
Running the audit without drama
Hold the session with the people who touch the pipeline daily, plus one reader of the output, ideally a planner or finance colleague who would be affected by a wrong figure. Ask each question aloud and demand evidence: open the repository, show the log, point to the monitor. Answers that begin with “I think” count as no until somebody shows the proof. Keep the tone curious; the goal is a ranked list of gaps, not a verdict on anyone’s work. Record the date and the scores so that the next pass has something to compare against.
Turning scores into a plan
Count your yes answers per section and rank the sections from weakest to strongest. The Atlas proposes a ninety-day starting plan that maps well onto the results. In the first month, list the critical outputs, put the code in version control, add the basic freshness, volume and key checks, and start the incident log. In the second month, add faster change testing, stage three critical tables before publication and write one producer contract. In the third, set freshness targets with an agreed allowance for misses, add lineage, tune seasonal monitors, write two more contracts and sample-audit a few loads. Adapt the numbers to your size; the order is the useful part.
Run the audit again after one quarter and compare section by section. The source for the framework is the Atlas’s DataOps entry, which includes the maturity ladder and a description of each practice in more depth than a checklist can carry. A small hospital team that completes one honest pass and fixes its weakest section has done more for data quality than one that buys a tool and skips the questions.
