Your phone buzzes at night for something that can wait, and soon you stop trusting pages. Here is how to reduce alert fatigue: page only on user-facing impact, group duplicates, route by severity, and review the noise often.
This guide sits inside the wider topic of error monitoring.
| Item | Value |
|---|---|
| Productive time lost to brief context switches | Up to 40% |
| Clinical alarms that are false (medical field) | 72% to 99% |
| Security alerts that were false (survey) | 52% |
| Security alerts that were redundant (survey) | 64% |
These figures come from New Relic’s January 2020 post and Last9’s guide to alert fatigue. They can change and come from different fields, so check the sources before you rely on them.
Why alert fatigue puts real incidents at risk
Alert fatigue is desensitization: when most pages are noise, people delay or skip the response. Atlassian’s post on alert fatigue describes the endgame: once faith is lost in alerting, you start to assume that every alert is false, and issues are missed.
Noise also costs focus. New Relic notes that research has shown people lose up to 40% of productive time with brief context switches, and even a quick “acknowledge” counts as one.
Other fields show the same pattern. Last9 cites research that 72% to 99% of clinical alarms are false, and a security survey where 52% of alerts were false and 64% were redundant.
New Relic’s post names the usual sources of noise:
- Irrelevant alerts from unused services or decommissioned projects
- Low-priority alerts that can wait
- Flapping alerts that fire and resolve repeatedly
- Duplicate alerts that repeat a problem you already know about
- Correlated alerts that are really one problem seen from many angles
Alert on user impact, not every system metric
Many teams inherit noise by switching on vendor defaults. OpenObserve’s write-up points out that these defaults are built to cover every possible customer, not your service or your traffic patterns.
A better starting point is user-facing signals, as Dr. Droid’s framework argues: start with user-facing signals and map those first. Examples from that post include:
- Failed user logins, not just authentication service uptime
- Checkout completion rates, not just payment gateway availability
- Slow API responses, not just average latency
- Database query failures, not just connection pool metrics
You can only alert on failures you can see, so the guide on how to track errors in production is the groundwork for everything here.
The most common mistake is paging on system metrics instead of user-facing symptoms. OpenObserve suggests asking whether users are actually experiencing degraded performance right now, and notes that system metrics are leading indicators at best and false alarms at worst. If response times are normal and error rates are flat, the page can usually wait for working hours.
Infrastructure alerts still matter, but they are indirect signals. Dr. Droid points out that a service can have high CPU usage while serving customers perfectly, so treat those readings as context for an investigation rather than an automatic reason to wake someone.
Last9 frames the same idea as a rule: don’t let yourself be bothered by alerts unless a certain proportion of users are affected by some metric degradation. Squadcast adds that you can collect most other metrics as data without alerting on them, and keep paging for the alerts that are core to your system reliability.
If you use service level objectives, a dual-window burn rate alert cuts false pages further. OpenObserve describes a short window of 5 minutes that catches sudden outages and a long window of 1 hour that confirms the burn is sustained, and both conditions must be true to fire the alert.
Choose objectives that customers would recognise. Atlassian notes that clients are usually interested in availability, error rate, request latency and system throughput, so those are stronger candidates for paging rules than internal numbers that mean nothing to users.
In practice, this means reviewing each paging rule and asking one plain question: if this fired right now and nobody acted, would a user notice? If the answer is no, move the rule to a dashboard, a ticket or a daytime channel, and keep the loud channels for symptoms that people can feel.
How do you tune thresholds and flapping alerts?
Start by watching what a noisy alert actually does. Squadcast gives an example of a CPU alert at 70% that regularly flaps between 69.8% and 70.8%, and suggests raising the value safely to 71% or even 72%.
Require a problem to be sustained before it pages anyone. Dr. Droid suggests alerting only after 3 consecutive failures, or requiring an issue to persist for 5 minutes. Other options from the same post:
- Use percentage-based thresholds rather than absolute counts
- Require state changes to persist before alerting
- Add cooldown periods between repeat notifications
Last9 recommends time buckets, so thresholds focus on errors over a larger period of time rather than single spikes. It also describes comparing the current error rate to the mean of previous days, plus or minus three times the standard deviation.
Group duplicates and route by severity
One underlying fault often fires many alerts. incident.io’s March 2025 post advises configuring your alerting system to group similar alerts within short periods automatically, and to enrich alerts with metadata so responders spend less time investigating.
For the mechanics of merging repeated errors into one issue, read about error grouping and deduplication. Squadcast adds that you should define deduplication rules so the on-call system merges duplicate events and notifies only the first time.
Then separate what needs a human now from what can wait. Rootly’s July 2026 post says only P0/P1 incidents should trigger a phone call during off-hours, while lower-priority events can go to Slack or a ticket for the next business day.
The same logic applies to thresholds on one resource. Last9 contrasts a database disk at 80 percent, a friendly reminder, with 90 percent, which needs immediate attention.
Ownership keeps the system clean. Dr. Droid argues that every alert must have a single owner, and that alerts without owners almost never get fixed.
How do you cut noise without missing real incidents?
Silencing is the tempting shortcut, but Dr. Droid notes that anyone can reduce alert noise by turning off alerts. Rootly warns of overcorrection, where aggressive filtering creates dangerous blind spots in observability.
A safer order of work:
- Confirm coverage first. Make sure user-facing failures are alerted on before you tune anything.
- Assign an owner and a runbook to each alert.
- Enrich the alert with context such as runbooks, dashboards and recent deployments.
- Tune thresholds and grouping using the techniques above.
- Review the data on a regular schedule and fix the worst offenders.
Measure the result. Rootly suggests regularly reviewing alerts per team, alerts acknowledged outside business hours, and MTTA, and incident.io suggests checking past incidents that were deferred to working hours or closed as not a real issue.
Frequently asked questions
What is alert fatigue?
Show answer
It is the desensitization that follows an excessive volume of alerts, which leads to delayed responses or missed critical notifications, as described by Last9.
Is muting noisy alerts a safe fix?
Show answer
Not on its own. Muting trades noise for blind spots, so confirm coverage of user-facing failures first and tune afterwards, as Dr. Droid advises.
Which alerts should wake someone up?
Show answer
Reserve phone calls for urgent, user-impacting problems. Send everything else to chat, email or a ticket for working hours.
How often should you review alerts?
Show answer
Make it a habit. Dr. Droid suggests reviewing an alert dashboard every sprint and fixing the top 3 worst offenders.
