Performing AI-Driven OSINT Correlation

When to Use

• You have collected raw OSINT data from multiple tools and sources but need to identify connections, contradictions, and patterns across them.
• You need to build a unified intelligence profile for a target entity (person, organization, or infrastructure) from fragmented data.
• Traditional manual correlation is too slow or error-prone for the volume of data collected.
• You want confidence-scored assessments of identity linkage across platforms rather than simple keyword matching.

Prerequisites

• Python 3.10+ with requests, json, and csv libraries
• Sherlock installed (pip install sherlock-project)
• theHarvester installed (pip install theHarvester)
• SpiderFoot 4.0+ running on localhost:5001
• Access to an LLM API (OpenAI, Anthropic, or local model via Ollama)
• Optional: Maltego CE for graph visualization of correlation results
• Optional: API keys for Shodan, VirusTotal, HaveIBeenPwned, Hunter.io

Workflow

Legal & Ethical Requirements

• Obtain documented written authorization before any investigation
• Establish lawful basis for data processing (law enforcement, corporate policy, etc.)
• Define PII retention limits and data handling procedures
• Comply with local privacy regulations (GDPR, CCPA, etc.)

Phase 1 — Multi-Source OSINT Collection

0. Create the working directory for all OSINT outputs:

   mkdir -p /tmp/osint
   

1. Enumerate usernames across platforms with Sherlock:

   sherlock "targetusername" --output /tmp/osint/sherlock-results.txt --csv
   

2. Harvest emails, subdomains, and hosts with theHarvester:

   theHarvester -d targetdomain.com -b all -f /tmp/osint/harvester-results.json
   

3. Run a SpiderFoot passive scan via REST API:

   curl -s http://localhost:5001/api/scan/start \
     -d "scanname=target-recon&scantarget=targetdomain.com&usecase=passive" \
     | jq '.scanid'
   

4. Export SpiderFoot results when scan completes:

   SCAN_ID="<scanid_from_step_3>"
   curl -s "http://localhost:5001/api/scan/${SCAN_ID}/results?type=all" \
     -o /tmp/osint/spiderfoot-results.json
   

5. Query breach databases for email exposure (example with HIBP API):

   curl -s -H "hibp-api-key: ${HIBP_KEY}" \
     -H "User-Agent: OSINT-Correlation-Skill" \
     "https://haveibeenpwned.com/api/v3/breachedaccount/target@example.com" \
     -o /tmp/osint/breach-results.json
   

Phase 2 — Data Normalization

6. Normalize all collected data into a common schema. Create a unified JSON structure that tags each finding with its source, timestamp, and data type:

   cat > /tmp/osint/normalize.py << 'EOF'
   import json, csv, sys, os
   from datetime import datetime
   
   findings = []
   
   # Normalize Sherlock CSV results
   sherlock_path = "/tmp/osint/sherlock-results.txt"
   if os.path.exists(sherlock_path):
       with open(sherlock_path) as f:
           for row in csv.DictReader(f):
               findings.append({
                   "source": "sherlock",
                   "type": "social_profile",
                   "platform": row.get("name", ""),
                   "url": row.get("url_user", ""),
                   "username": row.get("username", ""),
                   "status": row.get("status", ""),
                   "collected_at": datetime.utcnow().isoformat()
               })
   
   # Normalize theHarvester JSON results
   harvester_path = "/tmp/osint/harvester-results.json"
   if os.path.exists(harvester_path):
       with open(harvester_path) as f:
           data = json.load(f)
           for email in data.get("emails", []):
               findings.append({
                   "source": "theHarvester",
                   "type": "email",
                   "value": email,
                   "collected_at": datetime.utcnow().isoformat()
               })
           for host in data.get("hosts", []):
               findings.append({
                   "source": "theHarvester",
                   "type": "hostname",
                   "value": host,
                   "collected_at": datetime.utcnow().isoformat()
               })
   
   # Normalize SpiderFoot results
   sf_path = "/tmp/osint/spiderfoot-results.json"
   if os.path.exists(sf_path):
   

…