Triaging Security Alerts in Splunk

When to Use

Use this skill when:

• SOC Tier 1 analysts need to process the Incident Review queue in Splunk Enterprise Security (ES)
• Notable events require rapid severity classification and initial investigation before escalation
• Alert volume exceeds capacity and analysts need a systematic triage methodology
• Management requests metrics on alert disposition (true positive, false positive, benign)

Do not use for deep forensic investigation — escalate to Tier 2/3 after initial triage confirms malicious activity.

Prerequisites

• Splunk Enterprise Security 7.x+ with Incident Review dashboard configured
• CIM-normalized data sources (Windows Event Logs, firewall, proxy, endpoint)
• Role with ess_analyst capability for notable event status updates
• Familiarity with SPL (Search Processing Language)

Workflow

Step 1: Access Incident Review and Prioritize Queue

Open the Incident Review dashboard in Splunk ES. Sort notable events by urgency (calculated from severity x priority). Apply filters to focus on unassigned events:

| `notable`
| search status="new" OR status="unassigned"
| sort - urgency
| table _time, rule_name, src, dest, user, urgency, status
| head 50

Focus on Critical and High urgency events first. Group related alerts by src or dest to identify attack chains rather than treating each alert independently.

Step 2: Investigate the Notable Event Context

For each notable event, pivot to raw events. Example for a brute force alert:

index=wineventlog sourcetype="WinEventLog:Security" EventCode=4625
src_ip="192.168.1.105"
earliest=-1h latest=now
| stats count by src_ip, dest, user, status
| where count > 10
| sort - count

Check if the source IP is internal (lateral movement) or external (perimeter attack). Cross-reference with asset and identity lookups:

| `notable`
| search rule_name="Brute Force Access Behavior Detected"
| lookup asset_lookup_by_cidr ip AS src OUTPUT category, owner, priority
| lookup identity_lookup_expanded identity AS user OUTPUT department, managedBy
| table _time, src, dest, user, category, owner, department

Step 3: Correlate Across Data Sources

Check if the same source appears in other telemetry:

index=proxy OR index=firewall src="192.168.1.105" earliest=-24h
| stats count by index, sourcetype, action, dest_port
| sort - count

Look for corroborating evidence: Did the same IP also trigger DNS anomalies, proxy blocks, or endpoint detection alerts?

index=main sourcetype="cisco:asa" src="192.168.1.105" action=blocked earliest=-24h
| timechart span=1h count by dest_port

Step 4: Check Threat Intelligence Enrichment

Query the threat intelligence framework for known IOCs:

| `notable`
| search search_name="Threat - Threat Intelligence Match - Rule"
| lookup threat_intel_by_ip ip AS src OUTPUT threat_collection, threat_description, threat_key
| table _time, src, dest, threat_collection, threat_description, weight
| where weight >= 3

For domains, check against threat lists:

| tstats count from datamodel=Web where Web.url="*evil-domain.com*" by Web.src, Web.url, Web.status
| rename Web.* AS *

Step 5: Classify and Disposition the Alert

Update the notable event status in Incident Review:

Update via Splunk ES UI or REST API:

| sendalert update_notable_event param.status="2" param.urgency="critical"
  param.comment="Confirmed brute force from compromised workstation. Escalated to IR-2024-0431."
  param.owner="analyst_jdoe"

Step 6: Document Triage Findings

Record in the notable event comment field:

• Source/destination involved
• Data sources examined
• Correlation findings (related alerts, TI matches)
• Disposition rationale
• Next steps for escalation

| `notable`
| search rule_name="Brute Force*" status="closed"
| stats count by status_label, disposition

…