Implementing Alert Fatigue Reduction

When to Use

Use this skill when:

• SOC analysts face more alerts than they can reasonably investigate (>100 alerts/analyst/shift)
• False positive rates exceed 70% on key detection rules
• True positives are being missed or dismissed due to alert volume
• Management reports declining analyst morale or increasing turnover related to workload

Do not use to justify disabling detection rules without analysis — reducing alerts must not create detection blind spots.

Prerequisites

• SIEM with 90+ days of alert disposition data (true positive, false positive, benign)
• Alert metrics: volume, disposition rate, MTTD, MTTR per rule
• Detection engineering resources for rule tuning and testing
• Splunk ES with risk-based alerting (RBA) capability or equivalent
• Baseline analyst capacity metrics (alerts per analyst per shift)

Workflow

Step 1: Measure Current Alert Quality

Quantify the problem before making changes:

--- Alert volume and disposition analysis (last 90 days)
index=notable earliest=-90d
| stats count AS total_alerts,
        sum(eval(if(status_label="Resolved - True Positive", 1, 0))) AS true_positives,
        sum(eval(if(status_label="Resolved - False Positive", 1, 0))) AS false_positives,
        sum(eval(if(status_label="Resolved - Benign", 1, 0))) AS benign,
        sum(eval(if(status_label="New" OR status_label="In Progress", 1, 0))) AS unresolved
  by rule_name
| eval fp_rate = round(false_positives / total_alerts * 100, 1)
| eval tp_rate = round(true_positives / total_alerts * 100, 1)
| eval signal_to_noise = round(true_positives / (false_positives + 0.01), 2)
| sort - total_alerts
| table rule_name, total_alerts, true_positives, false_positives, benign, fp_rate, tp_rate, signal_to_noise

--- Top 10 noisiest rules (candidates for tuning)
| search fp_rate > 70 OR total_alerts > 1000
| sort - false_positives
| head 10

Daily alert volume per analyst:

index=notable earliest=-30d
| bin _time span=1d
| stats count AS daily_alerts by _time
| stats avg(daily_alerts) AS avg_daily, max(daily_alerts) AS peak_daily,
        stdev(daily_alerts) AS stdev_daily
| eval alerts_per_analyst = round(avg_daily / 6, 0)  --- 6 analysts per shift
| eval capacity_status = case(
    alerts_per_analyst > 100, "CRITICAL — Exceeds analyst capacity",
    alerts_per_analyst > 50, "WARNING — Approaching capacity limits",
    1=1, "HEALTHY — Within manageable range"
  )

Step 2: Implement Risk-Based Alerting (RBA)

Convert threshold-based alerts to risk scoring in Splunk ES:

--- Instead of generating an alert for every failed login, contribute risk
--- Risk Rule: Failed Authentication (contributes to risk score, no alert)
index=wineventlog EventCode=4625
| stats count by src_ip, TargetUserName, ComputerName
| where count > 5
| eval risk_score = case(
    count > 50, 40,
    count > 20, 25,
    count > 10, 15,
    count > 5, 5
  )
| eval risk_object = src_ip
| eval risk_object_type = "system"
| eval risk_message = count." failed logins from ".src_ip." targeting ".TargetUserName
| collect index=risk

--- Risk Rule: Successful Login After Failures (additive risk)
index=wineventlog EventCode=4624 Logon_Type=3
| lookup risk_scores src_ip AS src_ip OUTPUT total_risk
| where total_risk > 0
| eval risk_score = 30
| eval risk_message = "Successful login after ".total_risk." risk points from ".src_ip
| collect index=risk

--- Risk Threshold Alert: Only alert when cumulative risk exceeds threshold
index=risk earliest=-24h
| stats sum(risk_score) AS total_risk, values(risk_message) AS risk_events,
        dc(source) AS contributing_rules by risk_object
| where total_risk >= 75
| eval urgency = case(
    total_risk >= 150, "critical",
    total_risk >= 100, "high",
    total_risk >= 75, "medium"
  )
--- This single alert replaces 10+ individual threshold alerts

Before RBA vs After RBA comparison:

BEFORE RBA:
  Rule: "Failed Login > 5"         → 847 alerts/day  (FP rate: 92%)
  Rule: "Suspicious Process"       → 234 alerts/day  (FP rate: 78%)
  Rule: "Network Anomaly"          → 156 alerts/day  (FP rate: 85%)
  Total: 1,237 alerts/day

AFTER RBA:
  Risk aggregation alerts           → 23 alerts/day   (FP rate: 18%)
  Each alert contains full context from multiple risk contributions
  Reduction: 98% fewer alerts with HIGHER true positive rate

Step 3: Tune High-Volume False Positive Rules

…