At half past two on a Tuesday morning, the on-call phone belonging to a hospital’s systems engineer buzzed for the ninth time since midnight. Nine alerts, nine lookups, nine closures marked “nothing to see”. The engineer, whom we will call Marta (a composite, not a real person), had stopped reading the alert titles. She simply opened each one, scrolled to the bottom, and closed it. The next morning she told her manager the monitoring was broken. It was not broken in the sense that nothing fired. It was broken in the sense that nothing could be believed. This story is about diagnosing that condition and the fix that DetectionOps offers, told entirely from the technology and administration side.
The symptom: loud rules and quiet trust
When a monitoring estate has too many alerts, people describe the problem in different ways. The on-call engineer says it is exhausting. The manager says the dashboard is red every day. The security lead says the rules are fine because they were copied from a respected public collection. Nobody has a number. Marta’s team had hundreds of rules, no record of how often each one was right, and no way of knowing which ones had not triggered in a year because they were broken, rather than because nothing happened.
The first step in any diagnosis is to stop collecting opinions and start collecting evidence. In DetectionOps language, that means measuring each rule’s precision: of everything it flagged, how much was genuinely worth a person’s attention?
The examination: three weeks of counting
Marta’s group began by exporting every rule from the console into a repository, so that for the first time the whole set sat in one place. Then they added a spreadsheet column for each alert disposition: real, harmless but legitimate, or a mistake. After three weeks they had enough to compare rules.
The Ops Atlas explanation of DetectionOps measurement offers a worked illustration of what such a table can reveal. In its example, one rule raised forty alerts in a month, of which only six were genuine, giving a precision of fifteen percent, and every genuine hit swallowed about forty minutes of staff time. Marta’s table did not match those figures, because her estate was different, but it did show the same shape: a handful of rules produced most of the noise, and most rules were quiet and apparently fine.
One lesson from this stage deserves emphasis. The measurements were not sophisticated. A shared spreadsheet, a consistent three-way label and the discipline of filling it in were enough. The group did not need a new product to see that fewer than a fifth of the alerts from its worst rules were worth anyone’s time. What the exercise supplied was something the team had never had before: a shared factual basis for the argument. Arguments about monitoring tend to stall when each side relies on anecdotes, and they move quickly once a table of dispositions sits on the screen.
There was also a human benefit. Marta had privately wondered whether she was slow, or careless, because so many alerts felt pointless. The table showed her instinct had been right. Nine closures marked “nothing to see” had been the correct verdict nine times, and the problem lay in the rules, not in her.
The diagnosis: four separate causes
Once the data was in front of them, the group found that “too many alerts” was really four different conditions with four different remedies.
Cause one: rules built on indicators that expire
A number of rules matched specific file fingerprints or network addresses taken from old threat bulletins. In the model the Atlas refers to as the pyramid of pain, those sit on the lowest levels, because an adversary replaces them trivially and legitimate systems occasionally reuse the same addresses. These rules were both noisy and easy to evade.
Cause two: wholesale imports
The team had once adopted a large block of public rules in one go. The Atlas is clear that big community collections, which run to thousands of entries, are starting points to adapt, not packages to switch on. Many imported rules referred to log fields that Marta’s systems never produced, so they never fired; others fired on routine maintenance activity unique to a hospital environment.
Cause three: old exceptions nobody remembered
Over the years, engineers had silenced noisy patterns with filters. Each filter had been reasonable on the day it was added. None had an owner or an end date. The Atlas puts the principle bluntly: each filter trades a gap in visibility for sharper alerts, so the exemption should be recorded with evidence and an expiry. Marta’s team found several filters protecting systems that had been decommissioned and one that accidentally covered a whole subnet.
Cause four: fixes that did not stick
Twice in the previous year, someone had patched a rule straight inside the management screen to quiet a burst of pages. The next scheduled deployment overwrote the edit and the flood returned. Without a repository acting as the authoritative copy, the same fix had to be rediscovered.
The treatment plan
The remedy for each cause was different, and the order mattered. First, everything went into version control, and the console became read-only for rule edits. That stopped cause four on day one. Second, the team wrote a short test pair for each of the twenty noisiest rules, including one harmless case copied from real activity, and discovered that six of them could not pass their own “should stay quiet” test. Third, every filter received an owner and an expiry, and a quarter of them were removed immediately.
Fourth, the group retired the indicator-based rules that never produced a confirmed finding and replaced two of them with behaviour-based logic. Fifth, new rules entered a quiet trial period of roughly a week, in which they recorded what they would have flagged but paged nobody. Trial output was reviewed before anything reached the on-call phone. Sixth, a monthly scoreboard recorded precision, test completeness and minutes per genuine finding.
For readers who want the broader theme of modernising operational tooling without overwhelming the team, our article on streamlining software and cloud operations through engineering services discusses how to sequence that kind of work. And on the institutional side, the post about how hospital quality enters knee treatment decisions shows why reliable processes are worth the effort of documenting.
The follow-up visit
Eight weeks later, Marta’s phone buzzed far less. She could tell you why: the twenty worst rules had been tightened, the dead filters had been removed, and each remaining exemption had a reviewer’s name on it. More importantly, she trusted the alerts again. When one arrived at night, she read the title.
The team did not claim victory in terms of coverage. They knew from the Atlas that a label mapping rules to an attacker-behaviour catalogue is not evidence of detection. So they selected five behaviours that mattered most, simulated their harmless versions in a test environment, and recorded which ones produced an alert. The result was uncomfortable, with two of the five not detected at all, but it gave them a ranked list of work, which a vague sense of unease never had.
What to take from the case
- Noise is a symptom with several possible causes; count before you cure.
- Store the authoritative copy of every rule in a repository and forbid console-only edits.
- Test with a harmless example from your own environment, not just a textbook attack.
- Put an owner and an end date on every exemption.
- Prefer behaviour-based rules to single-indicator rules when the choice exists.
This account is a composite illustration, not a reported incident, and the numbers in the Atlas example belong to the Atlas example only. The method, though, travels well: measure first, treat causes separately, and give the on-call engineer a reason to trust the phone again.
