Detecting Deepfake Audio in Vishing Attacks

When to Use

• A suspected vishing call used an AI-cloned executive voice to authorize a wire transfer
• Security operations received a voicemail that sounds like the CEO but the tone seems off
• Incident response needs to determine whether a recorded phone call contains synthetic speech
• Fraud investigation requires forensic proof that audio was AI-generated
• Red team exercises use voice cloning and blue team needs detection capability

Do not use for text-based phishing (email/SMS); use email header analysis or URL detonation tools instead.

Prerequisites

• Python 3.9+ with librosa, numpy, scikit-learn, and scipy installed
• Audio samples in WAV, MP3, or FLAC format (mono or stereo, any sample rate)
• Reference corpus of known genuine voice samples for the targeted individual (optional but improves accuracy)
• FFmpeg installed for audio format conversion (librosa dependency)
• Minimum 3 seconds of audio for reliable feature extraction

Workflow

Step 1: Audio Preprocessing

Normalize and prepare audio samples for feature extraction:

import librosa
import numpy as np

# Load audio, resample to 16kHz mono
y, sr = librosa.load("suspect_call.wav", sr=16000, mono=True)

# Trim silence from beginning and end
y_trimmed, _ = librosa.effects.trim(y, top_db=25)

# Normalize amplitude to [-1, 1]
y_norm = y_trimmed / np.max(np.abs(y_trimmed))

Audio preprocessing ensures consistent feature extraction across different recording conditions, microphones, and codec artifacts.

Step 2: Extract Spectral Features

Extract the feature set that distinguishes real from synthetic speech:

Mel-Frequency Cepstral Coefficients (MFCCs):

# Extract 20 MFCCs + delta and delta-delta
mfccs = librosa.feature.mfcc(y=y_norm, sr=sr, n_mfcc=20)
mfcc_delta = librosa.feature.delta(mfccs)
mfcc_delta2 = librosa.feature.delta(mfccs, order=2)

MFCCs capture the spectral envelope of speech, representing how the vocal tract shapes sound. Deepfake audio often shows unnatural smoothness in higher-order MFCCs because neural vocoders approximate but do not perfectly replicate the acoustic resonance of a physical vocal tract.

Spectral Features:

spectral_centroid = librosa.feature.spectral_centroid(y=y_norm, sr=sr)
spectral_bandwidth = librosa.feature.spectral_bandwidth(y=y_norm, sr=sr)
spectral_contrast = librosa.feature.spectral_contrast(y=y_norm, sr=sr)
spectral_rolloff = librosa.feature.spectral_rolloff(y=y_norm, sr=sr)
zero_crossing_rate = librosa.feature.zero_crossing_rate(y_norm)

Key indicators of deepfake audio:

• Reduced spectral contrast in the 4-8 kHz range (vocoders compress high-frequency detail)
• Abnormally consistent spectral centroid over time (real speech has natural variation)
• Lower zero-crossing rate variance (synthetic speech lacks micro-perturbations)
• Missing or attenuated formant transitions during consonant-vowel boundaries

Step 3: Build Feature Vector and Classify

Aggregate frame-level features into a fixed-length vector and classify:

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

def build_feature_vector(y, sr):
    features = []
    mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=20)
    for coeff in mfccs:
        features.extend([np.mean(coeff), np.std(coeff), np.min(coeff), np.max(coeff)])
    for feat_fn in [librosa.feature.spectral_centroid,
                    librosa.feature.spectral_bandwidth,
                    librosa.feature.spectral_rolloff,
                    librosa.feature.zero_crossing_rate]:
        feat = feat_fn(y=y, sr=sr) if feat_fn != librosa.feature.zero_crossing_rate else feat_fn(y)
        features.extend([np.mean(feat), np.std(feat), np.min(feat), np.max(feat)])
    contrast = librosa.feature.spectral_contrast(y=y, sr=sr)
    for band in contrast:
        features.extend([np.mean(band), np.std(band)])
    return np.array(features)

Classification uses an ensemble approach: Random Forest for robustness and Gradient Boosting for accuracy, with a voting mechanism to reduce false positives.

Step 4: Temporal Artifact Analysis

Examine time-domain artifacts that neural vocoders leave behind:

# Pitch stability analysis - deepfakes often have unnaturally stable F0
f0, voiced_flag, voiced_probs = librosa.pyin(y_norm, fmin=50, fmax=500, sr=sr)

…