Data observability is supposed to tell you when something breaks. When it fires hundreds of alerts that nobody owns, people learn to ignore it. The fix is a short list of alerts, each tied to a metric that one person owns.
The channel nobody reads
You have probably seen this happen. A team turns on data observability. They add checks on every table they can find.
The first week, people read every alert. By the second month, the alert channel has hundreds of messages. Most of them are about tables nobody uses.
People mute the channel. Then a real problem fires, and it sits in the same pile as the noise. Your leaders find it first, in a report.
The tool did its job. It found the problem. Nobody was listening.
Why this is a data quality problem
It is easy to call this a tooling problem. I do not think it is.
An alert that nobody reads has the same effect as no alert. The bad data still gets to the report. The only difference is that you paid for a warning and then ignored it.
So alert fatigue is a data quality problem. It breaks the one thing monitoring is for: getting the right person to look at the right problem before it spreads.
More checks is not more coverage
When an alert gets missed, the first reaction is often to add more checks. That makes the noise worse.
Coverage is not the number of checks you run. Coverage is the number of problems that someone sees and acts on. If nobody acts, the check did not cover anything.
Tie every alert to an owned metric
In an earlier post, I wrote that nobody owns the metric in a lot of companies. Alerts have the same gap.
Start from the metrics, not the tables. For each metric your business depends on, ask three questions:
- Who owns this metric? One person, with a name.
- Which data paths feed it?
- What has to break on those paths for the number to be wrong?
Those answers give you your alert list. Each alert points at a path that feeds an owned metric. Each alert goes to that owner.
If you cannot name an owner, do not create the alert yet. Name the owner first.
Cut the list down
Then go through the alerts you already have. For each one, ask:
- Does it feed a metric someone owns?
- Did anyone act on it the last time it fired?
- Would anyone notice if it went away?
If the answer to all three is no, turn it off or move it to a log that nobody gets paged for. This step feels risky. It is the step that makes people trust the channel again.
Keep the list short. A short list that people read beats a long list that people mute.
Make the weekly review include the alerts
I wrote before that data quality is an operating cadence, not a cleanup project. The alert list belongs in that weekly review.
Add two questions to the agenda:
- Which alerts fired this week, and did the owner act on each one?
- Which alerts fired and nobody acted? Do we fix them, move them, or turn them off?
An alert that fires every week with no action is not a warning. It is noise you have not deleted yet.
Start small
You do not need a new tool for this. You need a list and some names.
Pick the metrics your leaders review each week. Name the owners. Keep only the alerts that protect those metrics, and send each one to its owner. Then add more only when an owner asks for it.