An on-call runbook is a short, step-by-step procedure for one specific alert: what it means, what to check first, how to fix the common causes, and when to hand off. A good one lets whoever holds the pager act in minutes, even if they have never touched the service. This guide gives you the structure, a copy-paste template, a worked example and the habits that keep runbooks from going stale.
TL;DR
- Write one runbook per alert, not one per system. Each starts from the symptom the pager reports and ends with either a fix or a named person to escalate to.
- Include the desired outcome, required access, ordered steps, error handling, escalation criteria, an owner and a validation check.
- Link the runbook from the alert itself, so the person paged is one click from the procedure.
- Automate only the steps that are safe to repeat, and keep a human in the loop for new or sensitive situations.
- Have a teammate run every runbook before you trust it, and review it after each incident that used it.
Table of Contents
- What an on-call runbook is, and how it differs from a playbook
- What every runbook should contain
- A runbook template you can copy
- Trigger
- Desired outcome
- Access and tools
- Diagnosis (in order)
- Fixes
- Validate recovery
- Escalate when
- Communicate
- After
- A worked example: the checkout endpoint is failing
- Connect the runbook to the alert
- What to automate, and what to keep human
- Keep runbooks current
- Our perspective on runbooks and alert quality
- Put your first runbooks on a monitor that pages you only when it matters
- FAQ
- Sources
What an on-call runbook is, and how it differs from a playbook
A runbook is a documented procedure for handling a specific operational situation: what to check, in what order, and what to do based on what you find. The scope is narrow on purpose. “API latency above threshold on the orders service” is a runbook. “A major outage” is bigger than a runbook can cover.
That bigger question belongs to a playbook, which covers roles, phases and communication for a whole category of situations. The usual split: a runbook says how to do this technical task, a playbook says how we organize ourselves when something big happens. Not everyone uses the words this way. TechTarget notes that organizations sometimes use the two terms interchangeably, so agree on your own definitions and write them at the top of your wiki.
If you want the playbook side, our incident response playbook guide covers a first-hour checklist. This article stays at the level of the individual alert.
What every runbook should contain
You do not need to invent a standard. The AWS Well-Architected Framework guidance on runbooks lists the minimum: step-by-step instructions, the desired outcome, the permissions and tools required, error handling and escalation guidance, an owner, a central location, and validation by having another team member execute it.
Translated into the sections of a page, that looks like this:
| Section | What it answers | Why it matters at 3 a.m. |
|---|---|---|
| Title and trigger | Which alert or symptom sends you here? | You find the right page in seconds |
| Outcome | What does “fixed” look like? | You know when to stop |
| Access and tools | What permissions, dashboards and CLIs do you need? | You discover a missing permission before the incident, not during |
| Diagnosis | What do you check, in order? | You do not improvise under stress |
| Remediation | What do you do for each likely cause? | The common fixes are one copy-paste away |
| Escalation | Who do you call, and when? | You hand off early instead of struggling alone |
| Validation | How do you confirm recovery? | You do not close an incident that is still broken |
| Owner and history | Who maintains it, and when did it last change? | You can trust the page or know who to ask |
TechTarget’s guidance on runbook templates adds two more standardization items: version history and a clear statement of the process’s goal. Both are cheap and both pay off the first time two people edit the same page.
An incident runbook, according to TechTarget, commonly identifies the issue, describes symptoms, gives resolution steps, specifies validation tests, defines escalation criteria and records reporting or follow-up requirements. That last item is worth keeping. A line that says “open a postmortem using this template” turns every incident into the input for the next improvement. Our incident postmortem template is a format you can paste straight into that step.
A runbook template you can copy
This is a plain-text skeleton for an SRE runbook. Paste it into your wiki or repository and fill it in for one alert at a time.
# Runbook: [Alert name exactly as it appears in the page]
Owner: [team or person]
Last reviewed: [date] Last validated by: [name]
Version history: [link or changelog]
## Trigger
This alert fires when: [condition, in plain words]
Customer impact: [what users see]
Severity: [level and who must be told]
## Desired outcome
[Service state that means this is resolved]
## Access and tools
- [Dashboards, with links]
- [CLI or console access, with the role needed]
- [Where credentials come from]
## Diagnosis (in order)
1. [Check, with exact command or link]
- If [result A]: go to Fix A
- If [result B]: go to Fix B
- If unclear: escalate
2. [Next check]
## Fixes
### Fix A: [probable cause]
[Steps, safe to run twice]
### Fix B: [probable cause]
[Steps]
## Validate recovery
- [Check that confirms the symptom is gone]
- [How long to watch before closing]
## Escalate when
- [Condition, for example: no improvement after 15 minutes]
- Contact: [name or rotation, and how to reach them]
## Communicate
- Post in [incident channel] using: [short update template]
- Update the status page if customers are affected
## After
- Open a postmortem if [criteria]
- Update this runbook with anything that was missing
Two details deserve emphasis. First, the “safe to run twice” note. AWS security incident response guidance recommends making response mechanisms safe to run more than once and in environments whose state you do not know. A tired engineer will repeat steps, so write steps that tolerate it.
Second, the escalation criteria. Put a number or a condition there, not “if needed.” A rule like “no improvement after 15 minutes” removes the social cost of asking for help.
A worked example: the checkout endpoint is failing
Here is how the template looks for a realistic alert. The names and thresholds are illustrative, not from a real system.
Runbook: Checkout health check failing
Trigger. An HTTP check on the checkout endpoint has failed several times in a row and the failure is confirmed from a second region. Customers cannot complete purchases.
Outcome. The endpoint returns a successful response from all monitored regions, and the order queue is draining.
Diagnosis.
- Open the monitor and look at the check history. Is the failure in one region or all of them? A single region points toward routing, a CDN or a regional dependency. All regions point toward your application or its database.
- Check the most recent deployment. If one shipped in the last hour, roll it back first and investigate afterward.
- Check the database connection pool and the payment provider’s status page.
- Look at error responses: a timeout and a 500 call for different fixes.
Fixes.
- Recent deploy: roll back, confirm recovery, tag the release for review.
- Pool exhausted: restart the affected workers, then check for a slow query.
- Third-party outage: enable the fallback message, post a status update, wait.
Escalate to the service owner if the rollback does not restore service within 15 minutes, or immediately if the database is involved.
Notice that step one, region by region, is a symptom-based question. The Google SRE incident management guide recommends alerting on symptoms, the end-to-end customer experience, rather than internal causes, so that notifications are actionable. Your runbook then walks from that symptom toward the cause.
Connect the runbook to the alert
A runbook nobody can find is a document, not a tool. The most useful habit is to put the runbook link in the alert message itself. Guidance such as Stéphane Robert’s incident runbook guide suggests linking runbooks directly from monitoring alerts, basing escalation on service severity and including communication templates, such as incident channel updates.
Three things make the link work in practice:
- Name the runbook after the alert. If the alert is called “Checkout health check failing,” the page has the same title. Search then works even when the link is missing.
- Keep the alert trustworthy. A runbook is only as good as the page that sends you to it. If the pager fires on every network blip, engineers stop opening runbooks and start muting alerts. We wrote about this in how to reduce alert noise without missing real incidents.
- Match the check to the symptom. A website check, a TCP port check and a cron heartbeat fail in different ways and need different first steps. If you run scheduled jobs, give each one its own runbook entry and see our notes on cron job monitoring. Certificates and domains deserve the same treatment, since an expiry is a predictable incident: certificate expiry monitoring covers what to watch.
Runbooks also shorten recovery time, but only if you measure it honestly. Our piece on mean time to recovery shows how one outage can produce three different numbers depending on where you start the clock.
What to automate, and what to keep human
Once a runbook has worked a dozen times, some steps become mechanical. Automate those. AWS advises creating automated mechanisms to triage and respond to common situations as incidents repeat, while keeping human responses for unique, new and sensitive incidents. AWS also says runbooks used for automated response should be connected to response plans so mitigation is consistent and repeatable.
A practical order for automating:
- Read-only diagnosis first. Gathering logs, check history and recent changes carries no risk and saves the first five minutes.
- Low-risk, reversible actions next. Restarting a stateless worker or pausing a noisy monitor during planned work.
- Anything destructive stays manual and behind a human decision.
Google’s SRE team describes the same ladder for AI-assisted operations: progressive authorization, least privilege, circuit breakers, dry runs and guardrails, with higher autonomy reserved for well-defined scenarios. The same source describes human-in-the-loop review for critical operations and full autonomy only for minor incidents. If you let an assistant touch production, give it the narrowest permissions that do the job.
AI assistants are also useful on the diagnosis side. With an assistant connected over MCP, you can ask it to read a monitor’s recent check history and compare regions instead of clicking through dashboards. We cover what to ask for in AI for monitoring: four checks on-call teams need first, and the broader workflow in AI incident response for on-call teams.
Keep runbooks current
An out-of-date runbook is worse than none, because it sends a stressed engineer down the wrong path with confidence. Three habits prevent that.
Validate before you trust. AWS recommends having another team member execute the runbook to validate it, then updating it through your normal change-management process. The author is the worst tester, because they already know the missing context.

Review after incidents. Every time a runbook is used, ask one question in the postmortem: what was missing, wrong or unclear? Fix it that day. Stéphane Robert’s guide also suggests quarterly reviews and game days, which gives you a rhythm for runbooks whose alerts rarely fire.
Make the goal measurable. The same guide says an effective runbook lets any on-call team member either solve or escalate the incident within a few minutes. During a game day, time how long a new teammate takes to reach that point. If it is slow, the runbook needs work, not the person.
A short checklist for each runbook review:
- Do the links and dashboards still exist?
- Do the commands still work with current permissions?
- Has the architecture changed since the last review?
- Is the owner still on the team?
- Did the last real incident reveal a step that is not written down?
Our perspective on runbooks and alert quality
A runbook begins with an alert, and the quality of that alert decides whether anyone follows the runbook. We have seen teams write careful procedures and then watch them go unused, because the pager had cried wolf too many times. The runbook was fine. The trust was gone.
That is why Uptime Beacon opens an incident only after several failures in a row, and on paid plans confirms the failure from a second location before anyone is paged. A runbook that starts with “this is a real, confirmed failure” can skip the first question every on-call engineer asks: is it actually down?
The second thing we care about is how fast you reach a diagnosis. Every plan, Free included, includes an MCP server. Connect Claude, ChatGPT, Cursor, VS Code, Codex or another MCP client, sign in, and the assistant can read your monitors, incidents and check history, and diagnose a failing monitor region by region. Turn on “Allow changes” for a team and it can also create, pause and resume monitors, acknowledge incidents and start maintenance windows. Nothing can be deleted over MCP. We built it that way because the most useful runbook steps are often reads, and the destructive ones should stay in human hands.
Our results are stored on a control plane and database in the EU, which matters when your runbooks and postmortems refer to monitoring data you are required to keep in one jurisdiction.
Aaditya Parashar
Put your first runbooks on a monitor that pages you only when it matters
Start with the three alerts that woke you most often last quarter and write one runbook for each. Then link them from the monitors that raise those alerts.
Uptime Beacon’s Free plan includes 5 monitors checked every 5 minutes from EU Central, one alert channel (email or Slack) and one status page, with no card required. The MCP server is included too, so you can try region-by-region diagnosis from your AI assistant on day one.
![]()
When you need more, Starter is €14.99 per month, Pro is €39.99 per month and Business is €89.99 per month. Paid plans add confirmation from a second location, and Starter and above include maintenance windows. Extra capacity is available as add-ons from €5 per month. See the details on the Uptime Beacon pricing page, or read how SaaS teams monitor APIs and cron jobs.
FAQ
What is the difference between a runbook and a playbook?
A runbook is a step-by-step procedure for a specific technical task or alert. A playbook is a higher-level guide for a category of situations, covering roles, phases and communication. Many teams use the words interchangeably, so define them in your own documentation.
How long should an on-call runbook be?
Short enough to follow under stress: one alert, one page. If a runbook needs scrolling through background theory, move the theory to a linked reference page and keep the procedure itself to ordered steps. A useful test is whether a teammate who did not write it can solve or escalate within a few minutes.
Where should runbooks live?
In one central location that everyone on the rotation can reach, including when your main systems are degraded. AWS guidance lists a central location as part of the minimum for a runbook. Link each page from the alert that triggers it, so nobody has to search.
How often should we review runbooks?
Review a runbook after every incident that used it, and schedule a periodic pass, for example quarterly, for the ones that rarely fire. A game day where a teammate follows the page exactly as written is the fastest way to find gaps.
Can an AI assistant run my runbook?
It can help with the read-only parts, such as pulling check history and comparing regions. For actions that change production, keep permissions narrow and require human approval for anything critical, as Google’s SRE guidance on progressive authorization describes.
Sources
- AWS Well-Architected: OPS07-BP03 Use runbooks to perform procedures
- AWS Systems Manager Automation runbooks with Incident Manager
- AWS Security Incident Response Guide
- Google SRE: Incident management guide
- Google SRE: AI and reliable operations
- TechTarget: Compare runbooks vs. playbooks for IT process documentation
- TechTarget: How to create a runbook template for uniform documentation
- Stéphane Robert: Runbooks and incident response



