Four-step image referring-expression pipeline: turns images plus KITTI bounding-box labels into region descriptions, scene captions, grounded referring expressions, and (optionally) verified expressions via VLM distillation. Use when the user wants to generate referring-expression annotations from i

Image Referring Expression Pipeline Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao setup skill first (host preflight, credentials, cross skill discovery). Generate referring expression and grounding annotations from images with KITTI format bounding box labels. A single VLM (Gemini or any OpenAI compatible endpoint) runs four steps: per object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double check verification pass. Purpose Transform (image, KITTI labels) pairs into a unified annotations.jsonl containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0 1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re examines those bboxes against the image and corrects mismatches. Pipeline Architecture Step 0 (region expr) — VLM emits one short discriminative phrase per KITTI bbox ( bbox 2d , type , color , description ). Step 1 (image caption) — VLM emits a holistic, location agnostic scene caption. Step 2 (grounding expr) — VLM groups Step 0 objects into grouping phrases and returns one bbox list per group, optionally using Step 1's caption as extra context. Step 3 (double check) — VLM re checks each Step 2 bbox against the image; bad matches are removed, slightly off boxes get tightened. Steps 0 and 1 run in parallel within a single thread pool (they only depend on the seed records). Each step writes its own step <N /annotations.jsonl and skips already processed images on re run unless workflow.force reprocess: true . Instructions Initial setup When a user wants to run this pipeline, walk through these steps: 1. Images : Ask for data.image dir , the directory containing .jpg , .jpeg , or .png images. 2. KITTI labels : Ask for data.kitti label dir , the directory containing one .txt label file per image. Each label line must use KITTI format: <type <truncated <occluded <alpha <bbox left <bbox top <bbox right <bbox bottom ... . Lines with fewer than 8 fields are silently skipped. Set this even for Step 1 only runs because Steps 0 and 2 require it. 3. Resume from existing annotations : If the user already has a unified annotations.jsonl from a previous run, set data.input annotations jsonl to that file instead of seeding from data.image dir and data.kitti label dir . 4. API access : Ask the user which VLM endpoint they want to use. Present these five options and act on the choice: 1. Gemini — set vlm.backend: "gemini" ; require GOOGLE API KEY (env var or vlm.gemini.api key ). 2. NIM (e.g. https://inference api.nvidia.com/v1 ) — set vlm.backend: "openai" ; collect base url , model name , and api key . 3. TAO inference microservice (self hosted, OpenAI compatible). Confirm whether the server is already running: Running — collect base url , model name , and (optionally) api key ; set vlm.backend: "openai" . Not running — guide the user through the skills/applications/tao run inference service skill, which stands up a local TAO inference microservice with an OpenAI compatible API. Before promising a specific model, check skills/applications/tao run inference service/references/service.yaml for valid network arch config basenames . Once the server is up, collect base url , model name , and (optionally) api key ; set vlm.backend: "openai" . 4. vLLM (self hosted, OpenAI compatible). Confirm whether the server is already running: Running — collect base url , model name , and (optionally) api key ; set vlm.backend: "openai" . Not running — follow [references/vllm server.md](references/vllm server.md) to install and launch a vLLM server, then collect base url , model name , and (optionally) api key ; set vlm.backend: "openai" . 5. Custom (any other OpenAI compatible endpoint) — set vlm.backend: "openai" ; collect base url , model name , and (optionally) api key . If the user has no endpoint and does not want to set one up, stop and help resolve API access first. 5. Workflow steps : Choose one of: Full pipeline: ["0", "1", "2", "3"] No caption generation: ["0", "2", "3"] , where Step 2 falls back to image only context No verification: ["0", "1", "2"] Custom subset: any supported subset of steps 6. Output format : Choose one of: jsonl : unified schema only legacy : byte compatible .txt.stepN files only both : writes both formats and is the default for downstream tooling Running the pipeline The pipeline runs inside the TAO Toolkit container via the auto label CLI: Generate a default spec: auto label default specs results dir=/results module name=auto label , then set autolabel type: "image referring expression" . All fields support Hydra dot notation overrides on the command line. See [references/configuration.md](references/configuration.md) for the full YAML structure, all parameters, model/endpoint setup, and error patterns. Recommended pilot workflow 1. Run on 5 10 images with all four steps. 2. Inspect step 0 region expr/annotations.jsonl — are object types, colors, and discriminating phrases accurate? 3. Inspect step 2 grounding expr/annotations.jsonl — are objects grouped sensibly, and do bbox coordinates match the described groups? 4. Inspect step 3 double check/annotations.jsonl — were mismatched bboxes removed or tightened? Are any new errors introduced (rare)? 5. If quality is insufficient, switch the VLM to a stronger model (e.g. gemini 2.5 pro or a larger Qwen3 VL endpoint), raise media resolution / max output tokens , then re run with workflow.force reprocess=true . 6. Scale to the full dataset once satisfied. Configuration Key configuration fields (full reference in [references/configuration.md](references/configuration.md)): Field Default Description workflow.steps ["0","1","2","3"] Which steps to execute ( 0 =region expr, 1 =image caption, 2 =grounding expr, 3 =double check) workflow.max workers 4 Parallel threads per step (watch API rate limits) workflow.force reprocess false Ignore cached per step outputs and reprocess from scratch workflow.output format "jsonl" (set to "both" in the default spec) "jsonl" , "legacy" , or "both" vlm.backend "gemini" "gemini" or "openai" (OpenAI compatible endpoint) data.image dir required Directory of input images ( .jpg / .jpeg / .png ) data.kitti label dir required (unless resuming) Directory of KITTI format .txt label files data.input annotations jsonl "" Optional pre seeded annotations.jsonl (skips KITTI seeding) Inputs Two ways to seed the pipeline: 1. Image directory + KITTI labels (default). Set data.image dir and data.kitti label dir . The orchestrator walks the image directory, reads the matching <stem .txt KITTI file, parses bboxes (fields 0 + 4 7), reads each image's width / height via PIL, and writes a seed annotations.jsonl to results dir/ . 2. Pre seeded annotations JSONL (resume / pre computed regions). Set data.input annotations jsonl to a file with one {"image id", "image path", "width", "height", "kitti bboxes": [...]} object per line. Outputs All outputs go to results dir/ : seed annotations.jsonl — initial per image records (unless input annotations jsonl was supplied). step 0 region expr/annotations.jsonl — adds regions[] (each with bbox / bbox 2d , type , color , description ). step 1 image caption/annotations.jsonl — adds caption (string). step 2 grounding expr/annotations.jsonl — adds expressions[] (each {text, instances: [{bbox: [x1,y1,x2,y2]}]} ). step 3 double check/annotations.jsonl — same shape as Step 2, with bboxes removed/updated. results dir/annotations.jsonl — copy of the last completed step's output. When workflow.output format is "legacy" or "both" , each step also writes byte compatible step <N /labels/<stem .txt.stepN files for the original 2d data engine tooling. Prerequisites Container : nvcr.io/nvidia/tao/tao toolkit:7.1.0 pyt <! versions key: images.tao toolkit.pyt API access : At least one VLM endpoint (Gemini API key or OpenAI compatible endpoint capable of image input) PIL / Pillow : Required to read image dimensions during seeding (already present in the TAO container)

tao-generate-referring-expressions