Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs.

Spark Environment Setup DGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121 GPU, 128GB unified memory, CUDA 13. This is a narrower and younger platform than a standard x86 CUDA 12 box, so package selection and ABI matching matter more than usual — the wheel ecosystem for aarch64 + CUDA 13 is still filling in. When to Use This Skill Setting up a fresh Spark box for training or inference. Hitting an import error mentioning libcudart , a missing symbol, or a wheel that "installed fine but won't load." A framework install (PyTorch, Unsloth, TRL, vLLM, xformers) fails, hangs, or silently falls back to CPU. Deciding whether to use an NGC container or bare pip. Restoring a working setup after an OS reinstall or a base image update, needing to re verify from scratch. Each of these accepts the same general fix: match the container/wheel combination to CUDA 13 and SM121, don't fight the ABI. Container First Rule Quick decision, before the detail below: Standard training/inference work → NGC PyTorch container. Unsloth centric fine tuning → Unsloth container (it ships the pinned Triton/xformers/transformers combination already validated for that path). Neither fits (custom system package, local IDE interpreter) → bare pip, following the exact sequence further down. Default to a container. Use nvcr.io/nvidia/pytorch:25.09 py3 as the base for general work — the newest tag confirmed working on this hardware; pull a newer blessed tag if locally available rather than hard blocking on 25.11 py3 . NGC's tag is dated, so running it directly is fine: unsloth/unsloth:dgxspark latest is a moving tag by contrast — resolve and pin its digest before running it for anything reproducible; the bare tag is a discovery step only, not the default invocation. Full pull inspect pin sequence and flag rationale/volume mounts for finetuning/ run dirs: references/container workflow.md . Treat bare pip as the exception. The reason for the container first stance is pinning, not convenience. Triton, xformers, and transformers versions interact narrowly with GB10's SM121 target and CUDA 13; a container locks all of them together against a combination already validated on this hardware. Bare pip leaves that resolution to you, one broken import at a time. When bare pip is warranted, follow the NVIDIA playbook's install sequence verbatim and in order: The second command's no deps flag is not optional — letting pip re resolve Unsloth's dependency tree on aarch64 is a common way to pull in an incompatible torch or triton build. The third line is not optional either: the NGC base image's bundled torchao is too old for current peft 's LoRA attach path ( ImportError: ... torchao ... only versions above 0.16.0 are supported ) — a hard blocker, not a warning. Every == pin above is load bearing, taken from the dated known good version matrix in references/stack matrix.md (its Last verified date governs staleness) — an unpinned install resolves current PyPI versions well outside what this Unsloth release supports. Pull a fresh tag when a new blessed release is announced. Rebuild locally from one of the two bases only when a project needs an extra system package layered in — not to "upgrade" a component the image already pins. Details on both paths: references/container workflow.md . One more preflight: official DGX Spark playbooks have shipped broken before. Check recent issues on github.com/NVIDIA/dgx spark playbooks (and the other resources in references/stack matrix.md ) before trusting a recipe verbatim for a long run. The ABI Rule The single most common failure on Spark is a CUDA 12/13 ABI mismatch: a wheel built against libcudart.so.12 loaded on a system that only has libcudart.so.13 . The install usually succeeds; the failure surfaces later as a missing symbol error or a segfault that doesn't obviously point at CUDA. Fix: pull wheels from download.pytorch.org/whl/cu130 (the cu130 tagged aarch64 builds), or use one of the containers above, which already carry a matched build. Before chasing a stack trace that mentions a CUDA symbol, check which CUDA tag the installed wheel was built against: If that output doesn't start with 13 , the ABI mismatch is the first thing to fix. NGC container builds (e.g. nvcr.io/nvidia/pytorch:25.09 py3 ) build torch internally against CUDA 13 with no +cu130 wheel tag — pip show torch won't say cu130 there, and that absence alone is not a failure. Typical symptoms: ImportError: undefined symbol referencing a CUDA runtime function. A segfault on the first .cuda() call, no useful traceback. A wheel that installs cleanly, then fails at import time — pip's resolver doesn't check CUDA ABI, only version constraints. Two "identical" environments behaving differently — usually one has a cu130 wheel, the other a cu121/cu124 leftover. The fix is the same regardless of symptom: match the wheel's CUDA tag to the system, or use a container that already does. Component Quick Table Condensed status for the components most likely to come up. Full table with wheel URLs, build flags, the sm 121 vs sm 121a distinction, and the dated known good version matrix: references/stack matrix.md . Component Status PyTorch ✅ official cu130 aarch64 wheels bitsandbytes ✅ works out of the box Triton ✅ needs the TRITON PTXAS PATH parameter set flash attn ❌ skip pip build; NGC bundles a working one — see spark training gotchas G2 xformers source build only ( TORCH CUDA ARCH LIST=12.1 ) vLLM nightly wheels only TransformerEngine / NVFP4 train container only Everything else — Unsloth, Axolotl, TRL, PEFT — installs cleanly through the container first path above. LLaMA Factory and NeMo are fragile on Spark; check upstream issues first. Verification Commands Confirm the environment can actually see the GPU before running anything expensive: This call returns two values; the exact output format is one line, <bool <cuda version : If it prints False instead, don't jump straight to a wheel reinstall — ABI mismatch is one cause among several: Hypothesis Quick check Runtime/flags nvidia smi fails in container too Device visibility echo $CUDA VISIBLE DEVICES Permissions ls l /dev/nvidia CUDA init state wedged process; retry fresh shell/container ABI mismatch (usual culprit) torch.version.cuda not 13.x Check nvidia smi first — if it doesn't show the GPU, it's one of the first three, not ABI. Reinstall a wheel only once ABI is confirmed. Per hypothesis detail: references/stack matrix.md . Run right after the container starts, before installing project specific packages. One more check: if Triton kernel compilation fails once training starts, set TRITON PTXAS PATH=/usr/local/cuda/bin/ptxas and retry — see references/stack matrix.md for the full workaround list. Next Steps A verified environment is only the starting point. See also: spark training gotchas for failure preflights before a training run, and spark memory thermal ops for unified memory OOMs and thermal throttling during long ones.

spark-environment-setup