You want to know when your app breaks, without a phone that buzzes all night. Here is how to set up error alerts: alert on user-facing symptoms, route by urgency, and give every alert an owner and a next step.
This guide sits inside the wider topic of error monitoring, which covers the habits and tools behind catching production errors.
| Item | Value |
|---|---|
| Average daily alerts at a security operations team | 4,484 alerts per day |
| Example symptom alert: checkout flow error rate | Above 1% |
| Example symptom alert: API latency | P95 above 500ms |
| Example basic threshold alert: error rate | Above 5% |
| Example spike alert: error rate | Doubles compared to 1 hour ago |
These figures come from PagerDuty’s alert fatigue guide and OneUptime’s alert rule design guide. The thresholds are examples, not universal values, and the pages can change, so check the sources before you rely on them.
Decide what deserves an alert
Start with a filter, not a tool. PagerDuty suggests three questions to sort every candidate alert, as described in its alert fatigue guide:
- Is the issue real? If not, no alert is required.
- Does the issue require attention? If not, log it for future reference.
- Is the issue urgent? If yes, use a high-priority channel for immediate response.
The reason to be strict is alert fatigue: a constant barrage of notifications makes people tune out. The same PagerDuty page says the average security operations team receives 4,484 daily alerts. That figure is for security teams, not app errors, but it shows how fast volume can swamp the people who need to react.
OneUptime’s alerting best practices put the test plainly: if the on-call engineer cannot take a specific action to resolve an alert, the alert should not exist. Its alert rule design guide adds that one well-tuned alert beats ten noisy ones.
Alert on symptoms, not internal causes
Users do not see your CPU graph. They see slow pages and failed checkouts, and OneUptime’s guidance is to alert on what users experience rather than on internal system metrics.
The alert rule design guide contrasts infrastructure state with user pain. Its pairs include these:
- Instead of “CPU > 80%”, alert on “API latency P95 > 500ms”.
- Instead of “Memory usage high”, alert on “Checkout flow error rate > 1%”.
For error alerts, two rule shapes cover most needs. A threshold rule fires when the error rate crosses a boundary, and the same guide’s example uses an error rate above 5%. A rate-of-change rule fires when the error rate doubles compared to 1 hour ago, which catches rapid increases even when absolute numbers are still low.
Do not guess the numbers. The guide says to use historical data to find the right values, so look at your normal error rate before you pick a threshold.
If you are unsure how individual exceptions get captured and grouped before an alert can fire, read about what is exception tracking first. Latency rules rely on performance data rather than error events, and the difference is explained in apm vs error monitoring.
Route alerts by urgency
Not every alert should reach a person the same way. PagerDuty’s guide sorts alerts into four types:
- Records are preserved in the monitoring system for historical context rather than requiring immediate action.
- Actionable alerts need a response and should go to the right team member.
- Low-urgency alerts go through non-disruptive channels, such as email or chat.
- High-urgency alerts, such as an outage, use instant channels like phone calls or SMS.
OneUptime’s best practices post shows the same idea as three severity levels. Critical pages the on-call person immediately, warning notifies them during work hours, and info creates a ticket. Its example configuration escalates a critical alert to the next person after 15 minutes and never auto-resolves it.
Separating non-urgent alerts so they are handled during regular work hours also lets your team rest for real emergencies, according to the PagerDuty guide.
How to set up error alerts with context and an owner
An alert that only says “CPU is high” leaves the responder guessing. OneUptime’s example of a good alert names the affected service, the cause and the first move: scale up or investigate a runaway process. Its sample rule also carries an impact note, a runbook link and remediation steps.
The alert rule design guide also says every alert needs an owner, because orphan alerts become noise. Put the owning team in the rule itself.
That guide also lays out a step-by-step approach to writing rules. Work through these steps for each new rule:
- Start with user impact. Ask what user-visible problem the alert detects.
- Choose the right signal. Different problems need different signal types, such as a fixed threshold or a rate of change.
- Set thresholds based on data. Check your historical numbers rather than guessing.
Once the rule exists, write its message so the responder can act without hunting for context: name the affected service, the likely cause and the first move. If you cannot say what the responder should do, the actionable test from earlier applies, and the alert should not exist.
Keep alert fatigue from creeping back
Alert setup is never finished. PagerDuty recommends examining your alert data, including how many alerts land during on and off-hours, and setting aside dedicated time each month or week to improve your workflows. Even a few hours can make a difference, per the same guide.
In each review, look at your most frequent alerts and ask whether each one is actionable. If not, reclassify it as a low-priority ticket instead of alerting team members. The guide notes that alerts without specific actions, like raw CPU or memory usage metrics, often add noise rather than value.
Watch for these warning signs that fatigue has set in:
- Delayed response times as people struggle to parse notifications.
- Missed alerts, when critical ones blend in with minor issues.
- False positives that people begin to ignore or mentally tune out.
Frequently asked questions
Which error alert should you create first?
Show answer
Start with a symptom users would feel, such as a checkout flow error rate above 1%, as in the example from OneUptime’s alert rule design guide. Set the number from your historical data, not a guess.
Should every error trigger a notification?
Show answer
No. Under PagerDuty’s sorting questions, an issue that is not real needs no alert, and one that needs no attention is logged for future reference.
How do you cut false alarms?
Show answer
Review your most frequent alerts on a regular schedule. Ask whether each is actionable, and reclassify the ones that are not as low-priority tickets, as the PagerDuty guide advises.
Where should alerts be sent?
Show answer
Match the channel to urgency. Low-urgency alerts can go to email or chat, while high-urgency ones such as an outage call for phone calls or SMS, per PagerDuty.
