Trains and fine-tunes vision models for object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3 — plus any Transformers classifier), and SAM/SAM2 segmentation using Hugging Face Transformers on Hugging Face Jobs cloud GPUs. C

Vision Model Training on Hugging Face Jobs Train object detection, image classification, and SAM/SAM2 segmentation models on managed cloud GPUs. No local GPU setup required—results are automatically saved to the Hugging Face Hub. When to Use This Skill Use this skill when users want to: Fine tune object detection models (D FINE, RT DETR v2, DETR, YOLOS) on cloud GPUs or local Fine tune image classification models (timm: MobileNetV3, MobileViT, ResNet, ViT/DINOv3, or any Transformers classifier) on cloud GPUs or local Fine tune SAM or SAM2 models for segmentation / image matting using bbox or point prompts Train bounding box detectors on custom datasets Train image classifiers on custom datasets Train segmentation models on custom mask datasets with prompts Run vision training jobs on Hugging Face Jobs infrastructure Ensure trained vision models are permanently saved to the Hub Related Skills hugging face jobs — General HF Jobs infrastructure: token authentication, hardware flavors, timeout management, cost estimation, secrets, environment variables, scheduled jobs, and result persistence. Refer to the Jobs skill for any non training specific Jobs questions (e.g., "how do secrets work?", "what hardware is available?", "how do I pass tokens?"). hugging face model trainer — TRL based language model training (SFT, DPO, GRPO). Use that skill for text/language model fine tuning. Local Script Execution Helper scripts use PEP 723 inline dependencies. Run them with uv run : Prerequisites Checklist Before starting any training job, verify: Account & Authentication Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan) Authenticated login: Check with hf whoami() (tool) or hf auth whoami (terminal) Token has write permissions MUST pass token in job secrets — see directive 3 below for syntax (MCP tool vs Python API) Dataset Requirements — Object Detection Dataset must exist on Hub Annotations must use the objects column with bbox , category (and optionally area ) sub fields Bboxes can be in xywh (COCO) or xyxy (Pascal VOC) format — auto detected and converted Categories can be integers or strings — strings are auto remapped to integer IDs image id column is optional — generated automatically if missing ALWAYS validate unknown datasets before GPU training (see Dataset Validation section) Dataset Requirements — Image Classification Dataset must exist on Hub Must have an image column (PIL images) and a label column (integer class IDs or strings) The label column can be ClassLabel type (with names) or plain integers/strings — strings are auto remapped Common column names auto detected: label , labels , class , fine label ALWAYS validate unknown datasets before GPU training (see Dataset Validation section) Dataset Requirements — SAM/SAM2 Segmentation Dataset must exist on Hub Must have an image column (PIL images) and a mask column (binary ground truth segmentation mask) Must have a prompt — either: A prompt column with JSON containing {"bbox": [x0,y0,x1,y1]} or {"point": [x,y]} OR a dedicated bbox column with [x0,y0,x1,y1] values OR a dedicated point column with [x,y] or [[x,y],...] values Bboxes should be in xyxy format (absolute pixel coordinates) Example dataset: merve/MicroMat mini (image matting with bbox prompts) ALWAYS validate unknown datasets before GPU training (see Dataset Validation section) Critical Settings Timeout must exceed expected training time — Default 30min is TOO SHORT. See directive 6 for recommended values. Hub push must be enabled — push to hub=True , hub model id="username/model name" , token in secrets Dataset Validation Validate dataset format BEFORE launching GPU training to prevent the 1 cause of training failures: format mismatches. ALWAYS validate for unknown/custom datasets or any dataset you haven't trained with before. Skip for cppe 5 (the default in the training script). Running the Inspector Option 1: Via HF Jobs (recommended — avoids local SSL/dependency issues): Option 2: Locally: Option 3: Via HfApi().run uv job() (if hf jobs MCP unavailable): Reading Results ✓ READY — Dataset is compatible, use directly ✗ NEEDS FORMATTING — Needs preprocessing (mapping code provided in output) Automatic Bbox Preprocessing The object detection training script ( scripts/object detection training.py ) automatically handles bbox format detection (xyxy→xywh conversion), bbox sanitization, image id generation, string category→integer remapping, and dataset truncation. No manual preprocessing needed — just ensure the dataset has objects.bbox and objects.category columns. Training workflow Copy this checklist and track progress: Step 1: Verify prerequisites Follow the Prerequisites Checklist above. Step 2: Validate dataset Run the dataset inspector BEFORE spending GPU time. See "Dataset Validation" section above. Step 3: Ask user preferences ALWAYS use the AskUserQuestion tool with option style format: Step 4: Prepare training script For object detection, use [scripts/object detection training.py](scripts/object detection training.py) as the production ready template. For image classification, use [scripts/image classification training.py](scripts/image classification training.py). For SAM/SAM2 segmentation, use [scripts/sam segmentation training.py](scripts/sam segmentation training.py). All scripts use HfArgumentParser — all configuration is passed via CLI arguments in script args , NOT by editing Python variables. For timm model details, see [references/timm trainer.md](references/timm trainer.md). For SAM2 training details, see [references/finetune sam2 trainer.md](references/finetune sam2 trainer.md). Step 5: Save script, submit job, and report 1. Save the script locally to submitted jobs/ in the workspace root (create if needed) with a descriptive name like training <dataset <YYYYMMDD HHMMSS .py . Tell the user the path. 2. Submit using hf jobs MCP tool (preferred) or HfApi().run uv job() — see directive 1 for both methods. Pass all config via script args . 3. Report the job ID (from .id attribute), monitoring URL, Trackio dashboard ( https://huggingface.co/spaces/{username}/trackio ), expected time, and estimated cost. 4. Wait for user to request status checks — don't poll automatically. Training jobs run asynchronously and can take hours. Critical directives These rules prevent common failures. Follow them exactly. 1. Job submission: hf jobs MCP tool vs Python API hf jobs() is an MCP tool, NOT a Python function. Do NOT try to import it from huggingface hub . Call it as a tool: If hf jobs MCP tool is unavailable , use the Python API directly: Critical differences between the two methods: hf jobs MCP tool HfApi().run uv job() script param Python code string or URL (NOT local paths) File path to .py file (NOT content) Token in secrets "$HF TOKEN" (auto replaced) get token() (actual token value) Timeout format String ( "4h" ) Seconds ( 14400 ) Rules for both methods: The training script MUST include PEP 723 inline metadata with dependencies Do NOT use image or command parameters (those belong to run job() , not run uv job() ) 2. Authentication via job secrets + explicit hub token injection Job config MUST include the token in secrets — syntax depends on submission method (see table above). Training script requirement: The Transformers Trainer calls create repo(token=self.args.hub token) during init () when push to hub=True . The training script MUST inject HF TOKEN into training args.hub token AFTER parsing args but BEFORE creating the Trainer . The template scripts/object detection training.py already includes this: If you write a custom script, you MUST include this token injection before the Trainer(...) call. Do NOT call login() in custom scripts unless replicating the full pattern from scripts/object detection training.py Do NOT rely on implicit token resolution ( hub token=None ) — unreliable in Jobs See the hugging face jobs skill → Token Usage Guide for full details 3. JobInfo attribute Access the job identifier using .id (NOT .job id or .name — these don't exist): 4. Required training flags and HfArgumentParser boolean syntax scripts/object detection training.py uses HfArgumentParser — all config is passed via script args . Boolean arguments have two syntaxes: bool fields (e.g., push to hub , do train ): Use as bare flags ( push to hub ) or negate with no prefix ( no remove unused columns ) Optional[bool] fields (e.g., greater is better ): MUST pass explicit value ( greater is better True ). Bare greater is better causes error: expected one argument Required flags for object detection: Required flags for image classification: Required flags for SAM/SAM2 segmentation: 5. Timeout management Default 30 min is TOO SHORT for object detection. Set minimum 2 4 hours. Add 30% buffer for model loading, preprocessing, and Hub push. Scenario Timeout Quick test (100 200 images, 5 10 epochs) 1h Development (500 1K images, 15 20 epochs) 2 3h Production (1K 5K images, 30 epochs) 4 6h Large dataset (5K+ images) 6 12h 6. Trackio monitoring Trackio is always enabled in the object detection training script — it calls trackio.init() and trackio.finish() automatically. No need to pass report to trackio . The project name is taken from output dir and the run name from run name . For image classification, pass report to trackio in TrainingArguments . Dashboard at: https://huggingface.co/spaces/{username}/trackio Model & hardware selection Recommended object detection models Model Params Use case ustc community/dfine small coco 10.4M Best starting point — fast, cheap, SOTA quality PekingU/rtdetr v2 r18vd 20.2M Lightweight real time detector ustc community/dfine large coco 31.4M Higher accuracy, still efficient PekingU/rtdetr v2 r50vd 43M Strong real time baseline ustc community/dfine xlarge obj365 63.5M Best accuracy (pretrained on Objects365) PekingU/rtdetr v2 r101vd 76M Largest RT DETR v2 variant Start with ustc community/dfine small coco for fast iteration. Move to D FINE Large or RT DETR v2 R50 for better accuracy. Recommended image classification models All timm/ models work out of the box via AutoModelForImageClassification (loaded as TimmWrapperForImageClassification ). See [references/timm trainer.md](references/timm trainer.md) for details. Model Params Use case timm/mobilenetv3 small 100.lamb in1k 2.5M Ultra lightweight — mobile/edge, fastest training timm/mobilevit s.cvnets in1k 5.6M Mobile transformer — good accuracy/speed trade off timm/resnet50.a1 in1k 25.6M Strong CNN baseline — reliable, well studied timm/vit base patch16 dinov3.lvd1689m 86.6M Best accuracy — DINOv3 self supervised ViT Start with timm/mobilenetv3 small 100.lamb in1k for fast iteration. Move to timm/resnet50.a1 in1k or timm/vit base patch16 dinov3.lvd1689m for better accuracy. Recommended SAM/SAM2 segmentation models Model Params Use case facebook/sam2.1 hiera tiny 38.9M Fastest SAM2 — good for quick experiments facebook/sam2.1 hiera small 46.0M Best starting point — good quality/speed balance facebook/sam2.1 hiera base plus 80.8M Higher capacity for complex segmentation facebook/sam2.1 hiera large 224.4M Best SAM2 accuracy — requires more VRAM facebook

huggingface-vision-trainer