Debug AOTInductor (AOTI) errors and crashes. Use when encountering AOTI segfaults, device mismatch errors, constant loading failures, or runtime errors from aot_compile, aot_load, aoti_compile_and_package, or aoti_load_package.

AOTI Debugging Guide This skill helps diagnose and fix common AOTInductor issues. Error Pattern Routing Check the error message and route to the appropriate sub guide: Triton Index Out of Bounds If the error matches this pattern: → Follow the guide in triton index out of bounds.md All Other Errors Continue with the sections below. First Step: Always Check Device and Shape Matching For ANY AOTI error (segfault, exception, crash, wrong output), ALWAYS check these first: 1. Compile device == Load device : The model must be loaded on the same device type it was compiled on 2. Input devices match : Runtime inputs must be on the same device as the compiled model 3. Input shapes match : Runtime input shapes must match the shapes used during compilation (or satisfy dynamic shape constraints) If any of these don't match, you will get errors ranging from segfaults to exceptions to wrong outputs. Key Constraint: Device Type Matching AOTI requires compile and load to use the same device type. If you compile on CUDA, you must load on CUDA (device index can differ) If you compile on CPU, you must load on CPU Cross device loading (e.g., compile on GPU, load on CPU) is NOT supported Common Error Patterns 1. Device Mismatch Segfault Symptom : Segfault, exception, or crash during aot load() or model execution. Example error messages : The specified pointer resides on host memory and is not registered with any CUDA device Crash during constant loading in AOTInductorModelBase Expected out tensor to have device cuda:0, but got cpu instead Cause : Compile and load device types don't match (see "First Step" above). Solution : Ensure compile and load use the same device type. If compiled on CPU, load on CPU. If compiled on CUDA, load on CUDA. 2. Input Device Mismatch at Runtime Symptom : RuntimeError during model execution. Cause : Input device doesn't match compile device (see "First Step" above). Better Debugging : Run with AOTI RUNTIME CHECK INPUTS=1 for clearer errors. This flag validates all input properties including device type, dtype, sizes, and strides: This produces actionable error messages like: Debugging CUDA Illegal Memory Access (IMA) Errors If you encounter CUDA illegal memory access errors, follow this systematic approach: Step 1: Sanity Checks Before diving deep, try these debugging flags: These flags take effect at compilation time (at codegen time): AOTI RUNTIME CHECK INPUTS=1 checks if inputs satisfy the same guards used during compilation TORCHINDUCTOR NAN ASSERTS=1 adds codegen before and after each kernel to check for NaN Step 2: Pinpoint the CUDA IMA CUDA IMA errors can be non deterministic. Use these flags to trigger the error deterministically: These flags take effect at runtime: PYTORCH NO CUDA MEMORY CACHING=1 disables PyTorch's Caching Allocator, which allocates bigger buffers than needed immediately. This is usually why CUDA IMA errors are non deterministic. CUDA LAUNCH BLOCKING=1 forces kernels to launch one at a time. Without this, you get "CUDA kernel errors might be asynchronously reported" warnings since kernels launch asynchronously. Step 3: Identify Problematic Kernels with Intermediate Value Debugger Use the AOTI Intermediate Value Debugger to pinpoint the problematic kernel: This prints kernels one by one at runtime. Together with previous flags, this shows which kernel was launched right before the error. To inspect inputs to a specific kernel: If inputs to the kernel are unexpected, inspect the kernel that produces the bad input. Additional Debugging Tools Logging and Tracing tlparse / TORCH TRACE : Provides complete output codes and records guards used TORCH LOGS : Use TORCH LOGS="+inductor,output code" to see more PT2 internal logs TORCH SHOW CPP STACKTRACES : Set to 1 to see more stack traces Common Sources of Issues Dynamic shapes : Historically a source of many IMAs. Pay special attention when debugging dynamic shape scenarios. Custom ops : Especially when implemented in C++ with dynamic shapes. The meta function may need to be Symint'ified. API Notes Deprecated API Current API The new API stores device metadata in the package, so aoti load package() automatically uses the correct device type. You can only change the device index (e.g., cuda:0 vs cuda:1), not the device type . Environment Variables Summary Variable When Purpose AOTI RUNTIME CHECK INPUTS=1 Compile time Validate inputs match compilation guards TORCHINDUCTOR NAN ASSERTS=1 Compile time Check for NaN before/after kernels PYTORCH NO CUDA MEMORY CACHING=1 Runtime Make IMA errors deterministic CUDA LAUNCH BLOCKING=1 Runtime Force synchronous kernel launches AOT INDUCTOR DEBUG INTERMEDIATE VALUE PRINTER=3 Compile time Print kernels at runtime TORCH LOGS="+inductor,output code" Runtime See PT2 internal logs TORCH SHOW CPP STACKTRACES=1 Runtime Show C++ stack traces

aoti-debug