AWS Resource Health & Issue Diagnosis

This workflow analyzes a specific AWS resource to assess its health status, diagnose potential issues using CloudWatch logs and metrics, and develop a comprehensive remediation plan for any problems discovered.

Prerequisites

• AWS CLI configured and authenticated
• Target AWS resource identified (name, type, and optionally region/account)
• CloudWatch logging and metrics enabled on the target resource

Workflow Steps

Step 1: Get AWS Diagnostic Best Practices

Fetch https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ for monitoring and troubleshooting guidance to inform the diagnostic approach.

Step 2: Resource Discovery & Identification

Locate the target resource using the appropriate AWS CLI command for its type:

# EC2
aws ec2 describe-instances --filters "Name=tag:Name,Values=<name>"
# Lambda
aws lambda get-function --function-name <name>
# RDS
aws rds describe-db-instances --db-instance-identifier <name>
# ECS
aws ecs describe-services --cluster <cluster> --services <name>
# ALB
aws elbv2 describe-load-balancers --names <name>
# DynamoDB
aws dynamodb describe-table --table-name <name>
# SQS
aws sqs get-queue-attributes --queue-url <url> --attribute-names All
# API Gateway
aws apigatewayv2 get-apis

If multiple matches are found, prompt the user to specify region/account.

Step 3: Health Status Assessment

Run service-specific health checks:

# EC2
aws ec2 describe-instance-status --instance-ids <id>

# RDS
aws rds describe-db-instances --db-instance-identifier <name> \
  --query 'DBInstances[0].DBInstanceStatus'

# Lambda - error rate over 24h
aws cloudwatch get-metric-statistics --namespace AWS/Lambda \
  --metric-name Errors --dimensions Name=FunctionName,Value=<name> \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 --statistics Sum

# ECS
aws ecs describe-services --cluster <cluster> --services <name> \
  --query 'services[0].[status,runningCount,desiredCount,pendingCount]'

Key health indicators by service type:

• Lambda: Error rate, throttle rate, duration P99, concurrent executions
• RDS: CPU utilization, FreeStorageSpace, DatabaseConnections, ReadLatency/WriteLatency
• ECS: Running vs desired task count, task stop reason
• ALB: TargetResponseTime, HTTPCode_ELB_5XX_Count, UnHealthyHostCount
• SQS: ApproximateNumberOfMessagesNotVisible, ApproximateAgeOfOldestMessage
• DynamoDB: ConsumedReadCapacityUnits, ThrottledRequests, SuccessfulRequestLatency

Step 4: Log & Metrics Analysis

Find log groups and run CloudWatch Logs Insights queries:

# Find log groups
aws logs describe-log-groups --log-group-name-prefix /aws/<service>/<name>

# Start a query (last 24h errors)
aws logs start-query \
  --log-group-name /aws/lambda/<name> \
  --start-time $(date -u -d '24 hours ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'filter @message like /ERROR/ | stats count(*) as errorCount by bin(1h)'

# Get results
aws logs get-query-results --query-id <id>

# Lambda cold starts
aws logs start-query \
  --log-group-name /aws/lambda/<name> \
  --start-time $(date -u -d '24 hours ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'filter @type = "REPORT" | filter @initDuration > 0 | stats count() as coldStarts by bin(1h)'

# RDS Performance Insights (if enabled)
aws pi get-resource-metrics \
  --service-type RDS --identifier db:<identifier> \
  --metric-queries '[{"Metric":"db.load.avg"}]' \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period-in-seconds 3600

Identify: recurring error patterns, correlation with deployments (CloudTrail), performance trends, dependency failures.

Step 5: Issue Classification & Root Cause Analysis

Severity:

• Critical: Service unavailable, data loss, security incidents
• High: Performance degradation, error rates >5%, intermittent failures
• Medium: Warnings, suboptimal configuration, minor performance issues
• Low: Informational alerts, optimization opportunities

Root Cause Categories:

• Configuration Issues: wrong settings, missing env vars, IAM permission denials
• Resource Constraints: CPU/memory/disk limits, Lambda throttling, RDS connection exhaustion
• Network Issues: security group rules, VPC routing, DNS, NACLs

…