remote-gpu-trainer — Remote GPU Job Orchestration

Overview

Deploy and babysit long-running GPU jobs on rented boxes you don't own, across any platform, and get the result off the box before the meter or a preemption kills it. The core insight: you are a short-term tenant on someone else's machine — so the job is to detach the work, make the result outlive the instance, and stop the meter safely, not to provision a cluster.

This skill is platform-agnostic at the core, platform-specific at the edges: a fixed set of operating principles + a 6-phase lifecycle that hold everywhere, plus one profile per platform (profiles/<platform>.md) that owns every concrete path, proxy, billing verb, and spot semantic. Its defensible value is the union the big orchestrators skip: Chinese cgroup-isolated rentals + bare-SSH cheap boxes + the disk-budget / monitoring / teardown reality that is the job on metered hardware.

When to Use This Skill

Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a RENTED or remote instance they do not own — training, eval, ablation sweeps, batch inference, or large data processing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云/矩池云/Featurize/ 揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual): "远程 GPU 训练", "GPU 租赁", "GPU rental", "租卡", "spot 抢占", "spot preemption", "断点续训", "resumable training", "tmux 训练守护", "防 SSH 断线", "scp/rsync 上传", "多实例 ablation", "远程 GPU 监控", "省钱关机/销毁实例", "stop vs terminate billing", "checkpoint 磁盘满", "CUDA OOM/显存不足", "loss NaN/loss spike", "loss 不下降/不收敛", "overfit 单 batch", "FSDP/DeepSpeed 配置", "多卡训练 hang", "dataloader worker/数据增广 bug". NOT for purely local single-GPU training, in-instance multi-GPU DDP (use torchrun/accelerate), managed multi-cloud price-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).

When NOT to use — and what to use instead

This skill is for the blind spot those tools leave: AutoDL + Chinese platforms, bare SSH/Slurm/K8s rentals, and the operational gotchas (inode caps, mirror stalls, cgroup OOM, silent sync, spot grace windows, irreversible teardown) that survive whichever provisioner you use.

Operating principles (the WHY — 10 invariants)

These hold on every metered, isolated, rented GPU; only the paths/CLI change. One line each; the deep form with cross-platform nuance is in references/principles.md (read it before Phase 0).

1. Minimize paid wall-clock. The meter runs the whole time — smoke locally on CPU before renting, launch detached, release the instant verification passes.
2. Cheap checks before expensive compute. A 1–2 batch CPU smoke (logger off) kills import/config/shape/scale bugs for ~free. (Smoke content → verifying-dl-experiments.)
3. Trust artifacts you loaded, not log lines that claim success. "synced/saved/done" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process/artifact.
4. Know what survives stop vs destroy. Per platform, identify exactly which mount survives a stop and which survives a terminate — the data you need often lives on the volatile one. (The single biggest portability trap.)
5. Storage fails on the dimension you're not watching. Disk dies on inodes before bytes; the real hog hides in a symlinked cache; clean by value (keep tiny evidence, drop big scratch); monitor df -i, not just df -h.
6. Never mutate inputs under a live run. A running job holds its scripts in memory by byte-offset; overwriting one mid-run re-executes blocks. Version filenames.

…