# Incident runbooks and a postmortem library — what to do at 3 a.m., and what you learned Recipe No. 34, Work and teams. From The know.sh Cookbook: https://know.sh/cookbook/runbooks - For: the SRE lead at a SaaS company who keeps the on-call runbooks - You bring: the alert rules from your monitoring, runbooks scattered across a wiki and pinned messages, and the notes from your last few incidents - You get: a runbook shelf with a document per service and a finding per alert, and a postmortem shelf with a document per incident, linked both ways - Time: an afternoon for your assistant to draft the first five services, a week of evenings to check them, then an hour after each incident - Keep it: private while you work; share one document by a read-only link An alert fires at three in the morning. The page says *LedgerQueueLagHigh*, links to a wiki page last edited two years ago, and the first command on it names a cluster that no longer exists. Last spring's postmortem would have told the on-call engineer what to check first, but it is in a document nobody can find. This recipe keeps two shelves. *Runbooks* holds a document per service and a finding per alert, named exactly as the alert pages, with the same four parts every time: what it means, first checks, commands and escalation. *Postmortems* holds a document per incident: a timeline, the contributing factors as *Problem* findings, and the actions as *Suggestion* or *Decision*. A coding assistant with the know.sh plugin, such as Claude Code, drafts the runbooks from the alert rules in your repository, drafts each postmortem from your notes, links it to the runbook that should have caught it, and can lay the follow-up actions out as a campaign. You check every line in the editor before anyone is paged with it. know.sh does not page anyone, run commands or watch a metric. Your paging tool and your monitoring stay where they are; this is the reference they point to. ## What you will use - **Shelf**: Two shelves, *Runbooks* and *Postmortems*, so a search for a service finds both what to do and what has happened. - **Research document**: A runbook document per service, and a postmortem document per incident, titled with its incident number. - **Finding**: A finding per alert in the runbooks; in postmortems, *Problem* for contributing factors and *Suggestion* or *Decision* for actions. - **Your AI assistant**: Claude Code, working in your repository, drafts runbooks from the alert rules and postmortems from incident notes; Claude or ChatGPT can do the same from files you attach. - **The editor**: Where you correct the drafted commands and first checks, mark night-paging alerts **Key**, and turn agreed actions into *Decision*. - **Links between documents**: Each postmortem links to the runbook finding for its alert, and the runbook links back under "Seen in". - **Campaigns**: Follow-up actions as operations with dependencies, created by Claude Code from the agreed decisions. - **Look up**: On your own on-call weeks, type the alert name from the page into Look up and land on its runbook entry. - **Public link**: A link per runbook document, placed in the alert’s description so the page carries it. ## Method ### 1. Have your assistant draft a runbook per service Start `claude` in the repository that holds your alert rules, with the know.sh plugin installed, and export the old wiki pages into the same folder. Ask it to make two shelves, *Runbooks* and *Postmortems*, and on *Runbooks* a document for each service that pages: *ledger-worker*, *billing-api*, *auth-gateway*, *Postgres primary*. Start with the five that paged most last quarter. Ask for the same overview on every service: what it does, what it depends on, which dashboards to open (by name), which team owns it, and the escalation path. If you use Claude or ChatGPT instead, attach the rules and pages to the chat; the result is the same shape. ### 2. Get a finding per alert, named as it pages Ask for a finding per alert, titled with the alert's exact name from the rule: *LedgerQueueLagHigh*. At 3 a.m. the engineer opens the service's link from the page and finds the alert by the name they were paged with; on your own on-call weeks, **Look up** (⌘K) takes you straight to it. Under the title, use four bold headings every time: **What it means**, **First checks**, **Commands**, **Escalation**. Commands go in code blocks, read-only first, with a plain warning before any command that changes something: "Changes state: only after the checks above." Then press **Edit** and check every command yourself against the live system; the assistant drafted them from rules and old pages, not from experience. Mark the alerts that page at night as **Key**. ### 3. Keep secrets and customer data out of every command A runbook is read by whoever holds the link, so no command in it carries a credential. Refer to connection strings through the environment or the secret store (`psql "$LEDGER_RO_URL"`), and write where the value lives, not what it is. Use placeholders for anything about a customer: ``, never a real account. The same applies to example output in a finding: trim the log lines of email addresses, names and invoice numbers before they go into a finding. ### 4. Put the link where the page lands Press **Share** on each runbook document, copy the link when it appears (it is shown once), and paste it into the alert's description in your monitoring, so the page carries it. Leave it with no expiry (the default). Set a password on every runbook link and keep it in the on-call password manager entry; leave off the link altogether any runbook that describes a security weakness. Links are made per document, which is the right size here: one per service. If a link ends up somewhere it should not, **replace** it, and paste the new one into the alert rules. ### 5. Hand the incident notes to Claude Code After an incident, save the incident channel's notes as a text file and remove customer names, emails and account numbers first. Start `claude` in that folder and ask it to draft a postmortem on the *Postmortems* shelf, titled with the incident number and a plain description. Ask for a fixed shape: an overview with the summary and impact, a *Timeline* finding in UTC, each contributing factor as a *Problem*, and each proposed action as a *Suggestion*. Ask it to link the postmortem to the runbook finding for the alert, add a "Seen in" line to that finding, and file what it cannot tell from the notes as a *Question*. ### 6. Review it blamelessly, then fix the runbook Read the draft against the logs before the review. An assistant can misread a timestamp or a sequence; the logs win. Write about roles and systems, never people: "the on-call engineer restarted the consumer", not a name. In the review, agree the actions. In the editor, change each agreed *Suggestion* to a *Decision*, and delete the ones you dropped. Then go back to the runbook: if the first checks were wrong, rewrite them now. **Revisions** show what the assistant wrote, what you changed and when, which is a record in itself. ### 7. Lay the follow-ups out as a campaign Ask Claude Code to turn the postmortem's *Decision* findings into a campaign: one operation set per incident, an operation per action, each linking back to its finding. Dependencies say what waits on what: *Add a consumer lag alert* **blocks** *Tune autoscaling*. The graph view shows the order. Campaigns are a plan for one person and have no assignees; put the owning team in each operation's description, and keep the tickets in your tracker. Mark operations done as the tickets close. ### 8. Read across the postmortems Because every contributing factor is a finding title, you can search Look up (⌘K) for *retry* or *certificate* to list every incident where it recurs; the index gathers names such as *Postgres* and *LedgerQueueLagHigh*, with the incidents they appear in. Once a quarter, ask your assistant which factors recur across the *Postmortems* shelf and which runbooks still say nothing about them. That list is your reliability plan for the next quarter. ## Prompts to try A coding assistant with the know.sh plugin, such as Claude Code: > Read incident-notes.txt in this folder. Draft a blameless postmortem on my know.sh Postmortems shelf titled with the incident number: summary and impact as the overview, a UTC timeline finding, each contributing factor as a Problem, each proposed action as a Suggestion. Use roles, not names. Link it to the runbook finding for the alert that fired, and file anything the notes do not settle as a Question. A coding assistant with the know.sh plugin, such as Claude Code: > Read the Decision findings in the INC-2026-031 postmortem and create a know.sh campaign called “INC-2026-031 follow-ups” with one operation per decision, each linking back to its finding and naming the owning team in its description. Add blocks dependencies where one action has to land before another, and leave every operation as draft. A coding assistant with the know.sh plugin, such as Claude Code: > Compare the commands in the billing-api runbook in know.sh with the Kubernetes manifests in this repository. List every namespace, deployment name, label or flag the runbook uses that no longer exists. Do not edit the runbook; leave a proposed note on each command instead. Your assistant, connected to know.sh (Claude, ChatGPT or a local model): > Using know.sh, read every document on my Postmortems shelf. List the contributing factors that appear in more than one incident, citing each Problem finding, and say which documents on the Runbooks shelf do not mention them yet. ## Variations - Before your own on-call week, ask your assistant for a quiz on the runbooks for the services you know least, and take it in know.sh. Typed answers must match exactly, which suits alert names and read-only commands. - When a postmortem action changes the design, file it as a *Decision* finding in the service’s runbook document and link the two. - Keep security incidents in a private document with no link at all, and follow your security team’s process for who is told what. - For a new engineer joining the rotation, add an *On-call basics* document and link it from the [onboarding handbook](/cookbook/onboarding-handbook). ## Where it falls short - know.sh does not work offline. If the outage takes out your network or your sign-in, the runbooks for those failures need a copy somewhere that does not depend on them. - Only you can edit the runbooks. Other engineers send corrections to you; a team where everyone edits runbooks directly is better served by a wiki or the repository. - Nothing here watches metrics, pages anyone or runs a command. It is the reference your alerts point to. - Drafted runbooks are only as good as the rules and pages the assistant read, and smaller local models call tools less reliably. Check every command; Revisions show everything an assistant changed. - Links are per document, so there is no single link to the whole *Runbooks* shelf; each service’s link goes in that service’s alerts. ## A note on secrets, customer data and blame No credentials, tokens or connection strings go into any document, shared or not; alert descriptions get copied into chat and tickets, so treat a runbook link as readable by anyone. Strip customer names, emails and account numbers from incident notes before Claude Code reads them, or they will end up in your library. Postmortems here are blameless: roles and systems, not people. Security incidents follow your security team's process and stay off links. Assistants can make mistakes; check every timeline against the logs and every command before it goes into a runbook. Indexed under: Runbooks, Postmortems, On-call, Alerts, Blameless reviews, Contributing factors, Incident follow-ups.