Spark Training Gotchas

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

When to Use This Skill

• A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
• A run OOMs while nvidia-smi still shows headroom.
• Throughput degrades partway through a run that started fine.
• Before any multi-hour or multi-epoch job on GB10.
• Wiring two Sparks together, before picking a parallelism strategy.
• Choosing between FP8 and NVFP4 for a Spark-hosted run.

Common Issues Quick Reference

The Ten Gotchas

G1: CUDA 12/13 ABI Mismatch

• SYMPTOM: ImportError: undefined symbol naming a CUDA function, or a segfault on the first .cuda() call.
• CAUSE: most PyPI wheels link libcudart.so.12; Spark ships CUDA 13. pip never checks CUDA ABI, so it surfaces only at import or first kernel launch.
• CHECK: references/gotcha-checks.md G1 — the wheel's CUDA build tag.
• FIX: reinstall from download.pytorch.org/whl/cu130 or use a matched container.

G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect

• SYMPTOM: pip install flash-attn still fails/hangs. Unsloth may also silently train flash-attn over an explicitly requested SDPA.
• CAUSE: no aarch64/sm_121 wheel for bare pip — but NGC containers ship a working SM121 flash-attn, and Unsloth auto-prefers it, dropping attn_implementation="sdpa".
• CHECK: references/gotcha-checks.md G2 — is flash-attn already present and working.
• FIX: bare pip — skip flash-attn, use SDPA (unchanged). On NGC — the only reliable override is the monkeypatch in references/gotcha-checks.md G2.

G3: UMA OOM Below 128GB

• SYMPTOM: OOM during model load/training while nvidia-smi still reports free memory under the 128GB cap — or, on some setups, [N/A] outright instead of a number.
• CAUSE: mmap and the CUDA allocator double-count pages during safetensors load; QLoRA can OOM earlier than bf16 since dequantization adds transient allocs.
• CHECK: references/gotcha-checks.md G3 — read free -g and /proc/meminfo, not nvidia-smi.
• FIX: drop the page cache with sync; echo 3 > /proc/sys/vm/drop_caches — needs root, a between-run reset, not a mid-training step.

G4: Thermal Throttling

• SYMPTOM: throughput drops partway through a multi-hour run, or the box spontaneously reboots under sustained load.
• CAUSE: sustained power draw caps around 100W versus the 240W rated figure; long runs push into that ceiling and throttle or, sometimes, reboot.
• CHECK: references/gotcha-checks.md G4 — sample nvidia-smi --query-gpu=temperature.gpu,power.draw.
• FIX: if power plateaus under 240W while temperature climbs, treat throttling as the cause; improve cooling or cap run length.

G5: Bandwidth Ceiling

• SYMPTOM: memory-bound workloads, decode-heavy RL loops especially, plateau well below expected throughput.
• CAUSE: 273 GB/s is a spec ceiling, not sustained; measured bandwidth runs 180–192 GB/s.
• CHECK: references/gotcha-checks.md G5 — observed step time vs. the measured range, not spec.
• FIX: budget throughput from 180–192 GB/s; revise a plan built on the 273 GB/s figure.

G6: Global UMA Resource Contention

• SYMPTOM: a process's KV cache/weights get evicted mid-run silently, no OOM in its own logs.
• CAUSE: unified memory is one global pool; an uncapped or near-capacity process competes with anything else and can evict it. A small,

…