# How do I reduce the on-call load on my engineering team?

> Page only for symptoms that need a person now, give every alert one owner and a runbook, turn each incident into a change, and count the pages before and after

- Author: Ash Gaikwad (https://ashgaikwad.com)
- Published: 2024-07-26
- Topics: on-call, reliability, incident response, leadership
- Canonical: https://ashgaikwad.com/blogs/how-do-i-reduce-the-on-call-load-on-my-engineering-team

To reduce on-call load, treat it as an alert design and ownership problem before you treat it as a staffing problem. Page a person only for a symptom that users feel and that needs action now. Give every alert one owning team and a runbook, send everything else to a ticket or a dashboard, and make each incident end in a change that removes a future page. Then count pages per shift before and after, so you know it worked.

I am Ash, a Staff engineer in Toronto. On one platform I ran, I reduced on-call load by 80% by removing noisy alerts, routing each alert to the team that owns the service, and writing runbooks. I also designed the incident process (severity levels, roles and blameless post-mortems) and ran the post-mortems. This article is for a staff engineer or an engineering leader who wants to do the same, and it covers the order I would work in, what it costs and how to tell it worked.

![How I decide whether an alert may page someone. It has to pass three tests in order. Do users feel it, must someone act now, and does it have one owner and a runbook. If all three are yes, it pages the team that owns the service. Any no sends it to a ticket, a dashboard line or the bin, and nobody is woken.](/images/blogs/three-tests-before-an-alert-pages.svg)

## Start by measuring the load you have

You cannot cut what you have not counted. The Google SRE book chapter on [being on-call](https://sre.google/sre-book/being-on-call/) splits the load into quantity and quality. Quantity is the share of an engineer's time spent on call, and the chapter says that "no more than 25% can be spent on-call". Quality is the number of incidents in a shift, and it gives "the maximum number of incidents per day is 2 per 12-hour on-call shift" as its target.

Use those two ideas, but count your own data. Export the last 30 days of pages from your paging tool and tag each one with the alert name, the team that got it, the time of day and whether anyone took an action. That gives you four numbers: pages per shift, pages outside working hours, the share of pages that needed a human to act, and the alerts that fire most often. Fix the top of the last list first, because that is where the biggest drop for the least work is likely to be. Keep the data as your baseline.

## Page only for what a person must act on now

The chapter on [monitoring distributed systems](https://sre.google/sre-book/monitoring-distributed-systems/) was written by Rob Ewaschuk, and it is the best short statement of this rule that I know. It says that "Every page should be actionable." It also says that "Paging a human is a quite expensive use of an employee’s time", and that "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts".

The practical rule is to page on symptoms and keep causes for diagnosis. A symptom is something a user can feel, such as failed requests, slow responses or a stuck queue. A cause is something that might lead to one, such as high CPU, a restarted pod or a disk at 75%. For causes, the same chapter says to "only worry about very definite, very imminent causes". A disk that will be full in an hour is worth a page. A disk that will be full in a month is a ticket.

So every alert gets one of three destinations. A page is for something that needs a person now. A ticket is for something that needs a person this week. A dashboard line is for everything that helps when you are already investigating. The chapter is blunt about the common fourth destination, saying that "Email alerts are of very limited value and tend to easily become overrun with noise". An alert sent to a channel nobody must read is a dashboard line with extra steps, and I would delete it.

### Threshold alerts or burn-rate alerts

A threshold alert fires when a number crosses a line, for example an error rate above 2% for five minutes. The Google SRE Workbook chapter on [alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) examines that approach. It finds that the alert fires on any event that threatens the objective, "exhibiting good recall", but that "Precision is low: The alert fires on many events that do not threaten the SLO." That describes a noisy pager.

The workbook's answer is burn rate, which it defines as "how fast, relative to the SLO, the service consumes the error budget." You page when the budget is being used up quickly over a long window and a short one at once, so a brief blip does not page and a continuing problem does. For a 99.9% objective its recommended starting points are 2% of the budget used in one hour (page), 5% in six hours (page) and 10% in three days (ticket).

This has a cost that the shortcuts skip. You need a service level objective (SLO), a reliability target that someone agreed to, before you can burn it. That agreement is real work with product, so I would not make it step one. Remove the alerts that no one acts on first, keep one symptom alert for each service, and add burn rates for the services that matter most once the noise has dropped.

## Give every alert one owner and one action

Ownership is the biggest lever I know, and it is the one that tips from a technical change into a team change. An alert that goes to a shared channel or a central rota reaches people who cannot fix the cause. They acknowledge it, look for the owner and hand it on, and none of that fixes anything.

So route each alert to the team that owns the service, through your paging tool, and make the alert carry what the responder needs. That is who owns it, what is broken, how bad it is, and a link to a runbook with the first checks. Ewaschuk's chapter sets the test as "Every page response should require intelligence." If the runbook says "restart the worker" and nothing else, a machine should do that, or the alert should not exist.

Two costs come with this. The first is that teams who never carried a pager now carry one, and the central team that used to absorb every page stops being the shock absorber. That conversation is easier with the baseline data in hand. The second is that you will find services nobody owns. Hand that list to engineering leadership to settle.

## Agree on severity levels and incident roles

When an incident happens, the next load is confusion. Severity levels and roles remove most of it. The PagerDuty [severity levels](https://docs.pagerduty.com/ops-guides/incident-response-guide/severity-levels) page uses numbered levels "with the lower numbered severities being more urgent", and says that if you are unsure between two levels, "treat it as the higher one." I would write each level in terms of user impact, and tie it to who is woken. The top two levels page a person. The rest become tickets.

For roles, the Google SRE Workbook chapter on [incident response](https://sre.google/workbook/incident-response/) names three: "the Incident Commander (IC), Communications Lead (CL), and Operations or Ops Lead (OL)". The IC leads, the CL and OL report to the IC, and "By default, the IC assumes all roles that have not been delegated yet." So a small incident has one person in all three seats, and a large one splits them. The PagerDuty [incident commander](https://docs.pagerduty.com/ops-guides/incident-response-guide/incident-commander) page adds a rule that is easy to break. The commander "should not be performing any actions or remediations, checking graphs, or investigating logs."

The person who found the problem is no longer also writing updates and deciding who to pull in, and escalation follows a rule and not a panic. I designed this process on the platform I mentioned.

## Run post-mortems that end in changes

A post-mortem is where an incident turns into fewer pages, or into nothing. The Google SRE book chapter on [postmortem culture](https://sre.google/sre-book/postmortem-culture/) says that to be blameless a post-mortem "must focus on identifying the contributing causes of the incident without indicting any individual or team". People who fear blame hide details.

The part teams skip is the end. The same chapter says that "An unreviewed postmortem might as well never have existed", and its review questions include "Is the action plan appropriate and are resulting bug fixes at appropriate priority?" I sort the actions into three kinds. One removes a source of pages, by fixing the cause or deleting the alert. One shortens the next response, such as a runbook step, a dashboard or an automation. One widens detection when users found the problem before the alert did. Each action gets one owner and a place in a normal plan, and open ones are read out weekly. In my opinion a post-mortem with no scheduled change is only a meeting.

## What I would not do, and what it costs

I would not add people to the rota first. More people share the same noisy pages across more nights. It is fair once the alerts are fixed, and a poor substitute before.

I would not delete cause alerts without a symptom alert in their place. The workbook says threshold alerts have good recall, so removing them removes detection. Keep one symptom alert for each service before you prune its cause alerts, and watch how incidents are found for a few weeks afterwards.

I would not automate recovery before asking whether the page should exist, because an automatic restart for an alert nobody needs hides the noise and keeps the cost. And I would not chase page count alone, because you can lower it by going blind.

The costs are plain. Ownership moves work onto teams that did not have it. Runbooks and post-mortem actions take engineering time from other work. SLOs need agreement. Some deleted alerts will be wrong, so you need a way to notice a miss.

## How to tell it worked

Repeat the baseline each month. Pages per shift and pages outside working hours should fall. The share of pages that needed action should rise, because what is left is real. Add a guard against going blind, which is the number of incidents that customers or support found before an alert did. If it grows while pages shrink, you removed the wrong alerts.

My numbers came from one platform, so yours will differ. There, on-call load fell by 80%. On the same platform, mean time to recovery also fell by 80% after I added CI/CD guardrails and per-service dashboards and alerts. I read the two together, though I cannot prove that one caused the other.

## A plan for the first month

1. Export 30 days of pages and tag each one with its alert, team, time of day and whether anyone acted.
2. Delete or demote every alert that no one acted on, and keep one symptom alert for each service.
3. Give every service one owning team, route its alerts there, and hand the list of unowned services to leadership.
4. Put a runbook link in every paging alert, write the severity levels on one page and name the incident roles.
5. Hold the first blameless post-mortem with owned, scheduled actions, then repeat the baseline and share the change.

Ownership moves faster with a shared road. I wrote about that in [how to get engineering teams to adopt an internal developer platform](/blogs/how-do-i-get-engineering-teams-to-adopt-an-internal-developer-platform), and about the guardrails and dashboards that sit beside this work in [how to migrate many services to Kubernetes without downtime](/blogs/how-do-i-migrate-many-services-to-kubernetes-without-downtime). If you need to bring several teams along, start with [how to lead a technical initiative across several teams](/blogs/how-do-i-lead-a-technical-initiative-across-several-teams).

A quiet pager is a good sign, unless you deleted the wrong alerts, so keep counting.

## Frequently asked questions

### Should I add more people to the on-call rotation to reduce the load?

Fix the alerts first. More people spread the same pages over more nights without removing any. Once pages are actionable and owned, a larger rota is a fair way to share what remains.

### How many pages per shift is too many?

The Google SRE book gives a target of at most 2 incidents per 12-hour shift. Treat that as a reference for incidents, not pages, and set your own limit from your baseline.

### Should we page on symptoms or on causes?

Page on symptoms that users feel and keep causes for dashboards and tickets. The exception is a cause that is certain and about to hurt users, such as a disk that fills within the hour.

### Do I need SLOs before I fix my alerts?

No. Removing alerts nobody acts on, adding owners and writing runbooks need no SLO. Burn-rate alerting does need one, so add it for your most important services after the noise has dropped.

### Who should run an incident, the person who was paged?

By default yes. The Google SRE Workbook says the incident commander takes every role that has not been delegated. As the incident grows, hand out the other roles.
