Escrita de Postmortems

Guia completo para escrever postmortems eficazes e sem culpa, que impulsionam o aprendizado organizacional e previnem a recorrência de incidentes.

Não use esta habilidade quando

• A tarefa não estiver relacionada à escrita de postmortems
• Você precisar de um domínio ou ferramenta diferente fora deste escopo

Instruções

• Clarifique objetivos, restrições e entradas necessárias.
• Aplique as melhores práticas relevantes e valide os resultados.
• Forneça passos acionáveis e verificação.
• Se exemplos detalhados forem necessários, abra resources/implementation-playbook.md.

Use esta habilidade quando

• Conduzindo revisões pós-incidente
• Escrevendo documentos de postmortem
• Facilitando reuniões de postmortem sem culpa
• Identificando causas raiz e fatores contribuintes
• Criando itens de acompanhamento acionáveis
• Construindo uma cultura de aprendizado organizacional

Conceitos Principais

1. Cultura Sem Culpa

2. Gatilhos de Postmortem

• Incidentes SEV1 ou SEV2
• Interrupções voltadas para o cliente > 15 minutos
• Incidentes de perda de dados ou segurança
• Quase-acidentes que poderiam ter sido graves
• Modos de falha novos
• Incidentes que exigiram intervenção incomum

Início Rápido

Linha do Tempo do Postmortem

Day 0: Incident occurs
Day 1-2: Draft postmortem document
Day 3-5: Postmortem meeting
Day 5-7: Finalize document, create tickets
Week 2+: Action item completion
Quarterly: Review patterns across incidents

Modelos

Modelo 1: Postmortem Padrão

# Postmortem: [Incident Title]

**Date**: 2024-01-15
**Authors**: @alice, @bob
**Status**: Draft | In Review | Final
**Incident Severity**: SEV2
**Incident Duration**: 47 minutes

## Executive Summary

On January 15, 2024, the payment processing service experienced a 47-minute outage affecting approximately 12,000 customers. The root cause was a database connection pool exhaustion triggered by a configuration change in deployment v2.3.4. The incident was resolved by rolling back to v2.3.3 and increasing connection pool limits.

**Impact**:
- 12,000 customers unable to complete purchases
- Estimated revenue loss: $45,000
- 847 support tickets created
- No data loss or security implications

## Timeline (All times UTC)

| Time | Event |
|------|-------|
| 14:23 | Deployment v2.3.4 completed to production |
| 14:31 | First alert: `payment_error_rate > 5%` |
| 14:33 | On-call engineer @alice acknowledges alert |
| 14:35 | Initial investigation begins, error rate at 23% |
| 14:41 | Incident declared SEV2, @bob joins |
| 14:45 | Database connection exhaustion identified |
| 14:52 | Decision to rollback deployment |
| 14:58 | Rollback to v2.3.3 initiated |
| 15:10 | Rollback complete, error rate dropping |
| 15:18 | Service fully recovered, incident resolved |

## Root Cause Analysis

### What Happened

The v2.3.4 deployment included a change to the database query pattern that inadvertently removed connection pooling for a frequently-called endpoint. Each request opened a new database connection instead of reusing pooled connections.

### Why It Happened

1. **Proximate Cause**: Code change in `PaymentRepository.java` replaced pooled `DataSource` with direct `DriverManager.getConnection()` calls.

2. **Contributing Factors**:
   - Code review did not catch the connection handling change
   - No integration tests specifically for connection pool behavior
   - Staging environment has lower traffic, masking the issue
   - Database connection metrics alert threshold was too high (90%)

3. **5 Whys Analysis**:
   - Why did the service fail? → Database connections exhausted
   - Why were connections exhausted? → Each request opened new connection
   - Why did each request open new connection? → Code bypassed connection pool
   - Why did code bypass connection pool? → Developer unfamiliar with codebase patterns
   - Why was developer unfamiliar? → No documentation on connection management patterns

### System Diagram

[Cliente] → [Balanceador de Carga] → [Serviço de Pagamento] → [Banco de Dados] ↓ Pool de Conexões (quebrado) ↓ Conexões diretas (causa)

## Detecção

### O que funcionou

…