Runbook Creation

Create effective operational runbooks, standard operating procedures, and troubleshooting guides that any on-call engineer can follow under pressure.

Runbook Template — Full Structure

# Runbook: [Service / Process Name]

**Owner:** [Team or individual]
**Last Reviewed:** YYYY-MM-DD
**Version:** X.Y
**Severity if unavailable:** SEV[1-4]

---

## Overview

Brief description of the service, why this runbook exists, and when to
use it.

## Prerequisites

- [ ] Required access / IAM role: [details]
- [ ] Tools installed: [kubectl, aws-cli, psql, etc.]
- [ ] VPN connected to [environment]
- [ ] Communication channel open: [Slack #channel]

## Procedure

### Step 1 — [Action Name]

[Explanation of what this step does and why.]

```bash
# command here
```

**Expected output:** [describe what success looks like]

### Step 2 — [Action Name]

```bash
# command here
```

**Expected output:** [description]

*(Continue with numbered steps...)*

## Verification

How to confirm the procedure succeeded:

- [ ] [Check 1 — e.g., health endpoint returns 200]
- [ ] [Check 2 — e.g., no errors in logs for 5 minutes]
- [ ] [Check 3 — e.g., metrics return to baseline]

## Rollback

If the procedure fails or causes unexpected issues:

### Rollback Step 1
```bash
# rollback command
```

### Rollback Step 2
```bash
# rollback command
```

## Troubleshooting

| Symptom | Likely Cause | Resolution |
|---------|-------------|------------|
| [symptom 1] | [cause] | [fix] |
| [symptom 2] | [cause] | [fix] |

## Escalation

If unresolved after [X] minutes:
- **Primary:** @[team-lead] — [phone/Slack]
- **Secondary:** @[manager] — [phone/Slack]

## Related Runbooks

- [Link to related runbook 1]
- [Link to related runbook 2]

## Change Log

| Date | Author | Change |
|------|--------|--------|
| YYYY-MM-DD | [Name] | Initial version |

Overview

Failover the primary PostgreSQL instance to the synchronous replica when the primary is unreachable or degraded. This runbook covers both planned (maintenance) and unplanned (emergency) failover.

Prerequisites

• DBA or SRE-level access to primary and replica hosts
• psql client installed (v14+)
• VPN connected to production network
• Slack channel #db-ops open
• Confirm replica is in sync: replication lag < 1 MB

Procedure

Step 1 — Verify Replica Health

psql -h replica.db.internal -U dba -d postgres -c \
  "SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn();"

Expected output: pg_is_in_recovery = t, LSN advancing.

Step 2 — Stop Application Writes

kubectl scale deployment api-server --replicas=0 -n production
kubectl scale deployment worker --replicas=0 -n production

Expected output: Deployments scaled to 0 pods.

Step 3 — Confirm Write Quiesce

psql -h primary.db.internal -U dba -d postgres -c \
  "SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND query !~ 'pg_stat';"

Expected output: Count = 0 (no active queries).

Step 4 — Promote Replica

psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_promote();"

Wait up to 30 seconds, then confirm:

psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_is_in_recovery();"

Expected output: pg_is_in_recovery = f (no longer a replica).

Step 5 — Update DNS

aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "db.internal.example.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "replica.db.internal"}]
      }
    }]
  }'

Step 6 — Restart Application

kubectl scale deployment api-server --replicas=6 -n production
kubectl scale deployment worker --replicas=4 -n production

Verification

• psql -h db.internal.example.com -c "SELECT 1;" returns successfully
• Application logs show successful DB connections (no errors for 5 min)
• Transaction throughput returns to baseline on Grafana dashboard
• No replication-lag alerts firing

Rollback

If the promoted replica has issues, restore from the most recent backup:

# Restore latest automated snapshot (RDS example)
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier prod-db-restored \
  --db-snapshot-identifier prod-db-latest-snapshot

Escalation

…