Stereo depth estimation using FoundationStereo. Predicts disparity maps from stereo image pairs for 3D reconstruction. Use when training, evaluating, exporting, or running inference for a TAO FoundationStereo model. Trigger phrases include "train stereo depth", "FoundationStereo", "stereo disparity

Depth Net Stereo Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao setup skill first (host preflight, credentials, cross skill discovery). Stereo depth estimation using FoundationStereo architecture. Predicts disparity maps from stereo image pairs for 3D reconstruction. Uses pretrained Depth Anything v2 and EdgeNeXt encoders. Set model.stereo backbone.depth anything v2 pretrained path and model.stereo backbone.edgenext pretrained path . The mono and stereo skills both invoke the unified TAO depth net CLI inside the container; the mono/stereo family is selected via model.model type (e.g., FoundationStereo ). PyT actions packaged by this model skill: train , evaluate , inference , export , and quantize . The PyT depth net entrypoint does not accept a gen trt engine action in the current TAO image; build TensorRT engines only through the deploy workflow. For TAO Deploy TensorRT actions ( gen trt engine , TensorRT evaluate , and TensorRT inference ), read references/tao deploy foundation stereo.md first. The deploy spec template lives in this skill's references/spec template deploy.yaml . Train Action Policy This model is AutoML enabled at the model layer. Before handling any train stage request, read references/skill info.yaml and resolve the run override from either an explicit automl policy value or the user's workflow request. Use automl policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl policy: off for this run only. When automl policy: on , automl enabled: true , and both schemas/train.schema.json and references/spec template train.yaml are packaged, route the train action through tao skill bank:tao run automl by default with this model's skill dir . Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl policy . Use direct model training only when automl policy: off or the packaged train schema/template is missing; in the missing schema case, report that AutoML is enabled but not runnable for this model until schemas are generated. Non train actions such as evaluate , inference , export , and deploy flows stay in this model skill. The per run automl policy override does not change model metadata. Workflow Prerequisites — data accessibility Your dataset (left + right images + GT disparity) must be reachable from inside the container: SDK runner : place files at the S3 paths the runner resolves (the S3 TRAIN / S3 EVAL placeholders shown in the spec overrides). The runner handles S3 → container path mounting transparently. Direct docker run (e.g. local testing): mount the host dataset root read only at the same in container path: The same accessibility requirement applies to the <output dir written by all actions. Step 1 — Annotation file Per line annotation file referenced by data sources[ ].data file : Columns Format Use 2 <left <right Stereo inference (no GT) 3 <left <right <disparity Stereo with GT 4 <left <right <disparity <occlusion mask Stereo with GT and occlusion mask If you already have one, point to it. Otherwise generate via depth net convert : convert spec.yaml template (stereo): convert walks data root recursively, selects paths whose path string contains all substrings in image dir pattern (AND filter), then derives right / depth / mask paths by replacing image dir pattern[0] with the corresponding pattern's first element plus extension swap. Inspect your dataset's directory layout and identify the substrings distinguishing left, right, and GT (e.g. im0 vs im1 vs disp0GT for Middlebury). Step 2 — Pair model type and dataset name based on your data Prefer the dataset specific class when your layout matches a supported one — it applies class specific path conventions, evaluation crops, and (where applicable) occlusion mask handling. Fall back to GenericDataset only for layouts that do not match any registered class. Data category model type dataset name Middlebury data FoundationStereo Middlebury KITTI data FoundationStereo Kitti ETH3D data FoundationStereo Eth3d FSD synthetic data FoundationStereo FSD IsaacReal synthetic data FoundationStereo IsaacRealDataset Crestereo synthetic data FoundationStereo Crestereo Other / non canonical layout FoundationStereo GenericDataset Valid dataset name values for stereo data sources (case insensitive): FSD , IsaacRealDataset , Crestereo , Middlebury , Eth3d , Kitti , GenericDataset . The same dataset name value applies across train and evaluate actions (all of which use 3 column or 4 column annotations with GT disparity). The deploy side evaluate action follows the same rule — see references/tao deploy foundation stereo.md . For inference with 2 column annotations (left + right, no GT), use dataset name: GenericDataset regardless of data layout — the dataset specific classes ( Middlebury / Kitti / Eth3d / FSD / IsaacRealDataset / Crestereo ) require 3 column input and reject 2 column annotations at the dataloader level. For inference with 3 column annotations (left + right + GT), the dataset specific class is fine. Step 3 — Write spec yaml from the spec overrides Copy the action block from references/spec overrides foundation stereo.md . Replace: model.model type from Step 2 (typically FoundationStereo ) dataset.<... .data sources[ ].dataset name from Step 2 dataset.<... .data sources[ ].data file with the path from Step 1 For deploy side evaluate : enforce dataset.test dataset.batch size: 1 (see references/tao deploy foundation stereo.md ). Shape consistency: the crop size in dataset.test dataset.augmentation.crop size should match export.input height / input width so the trained model evaluator and the deploy side TensorRT evaluator operate at the same shape. Note that crop size is decorative on the pyt evaluate path but authoritative on the deploy evaluate side — see references/troubleshooting foundation stereo.md and references/tao deploy foundation stereo.md . Fresh install smoke runs are validated at crop size: [128, 128] with dataset.max disparity: 128 and model.max disparity: 128 . Avoid 112×112 crops and avoid setting max disparity smaller than the square crop side for smoke tests: those combinations can fail inside FoundationStereo with feature map or loss mask shape mismatches before a checkpoint is produced. Data source overrides are mandatory for every action . Each data sources entry is a dict with two mandatory fields: data file and dataset name . See references/spec overrides foundation stereo.md for the per action dataset requirements table, every action's override block, and the quantize known issue note. Step 4 — Run Create writable home/cache directories inside the mounted output path before using user . Some TAO containers do not have an /etc/passwd entry for the host UID, and PyTorch / matplotlib need writable cache paths when running as that UID. Without user "$(id u):$(id g)" the container writes outputs as nobody:nogroup , blocking host side cleanup / retry. Step 5 — Verify Container exit code 0 status.json kpi block populated For train : inspect per step train loss directly (the entrypoint reports Execution status: PASS even when loss is NaN) For evaluate : rely on epe / bp1 / bp2 / bp3 / d1 / rmse (the evaluator also emits abs rel / sq rel / rmse log which are non meaningful for stereo — see references/parameters foundation stereo.md ) For inference : artifacts under results dir For TAO Deploy TensorRT actions ( gen trt engine , TensorRT evaluate , and TensorRT inference ), read references/tao deploy foundation stereo.md first. Deploy spec templates live in this skill's references/ folder with the spec template deploy .yaml prefix. Training Requirements Monitoring metric: val/loss Eval dataset: optional. Val dataset configured via dataset.val dataset.data sources (each entry needs data file and dataset name ). See references/spec overrides foundation stereo.md for the per action dataset requirements table and every action's mandatory data source override block. Parameters, Metrics, Multi GPU, Export/TRT, Hardware See references/parameters foundation stereo.md for the full Important Parameters list (incl. model.encoder vits override, model.max disparity default 416, model.volume dim no op note, dataset.baseline , dataset.focal x , train.precision , export.batch size ), the Evaluation Metrics table, Multi GPU / Multi Node launch keys, Export / TRT Defaults ( opset version / on cpu pairing, NGC 576×960 settings), and Hardware requirements. Error Patterns and Troubleshooting See references/troubleshooting foundation stereo.md for disparity overflow, smoke test shape mismatch, missing pretrained paths, the encoder / dataset name struct errors, the depth net stereo: not found entrypoint note, the pyt vs deploy crop size discussion, and the deploy evaluate scalar conversion failure. Spec Param / Parent Model Inference See references/checkpoint inference mappings foundation stereo.md for the checkpoint resolution rules ( model epoch <epoch step <step .pth , dn model latest.pth policy), the absence of parent PyT gen trt engine , and the full per action inference mapping table from depth net stereo.config.json (including parent model / parent job id resolution). Deployment [tao deploy foundation stereo](references/tao deploy foundation stereo.md)

tao-train-foundation-stereo