GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cle

NeMo Curator GPU Accelerated Data Curation NVIDIA's toolkit for preparing high quality training data for LLMs. When to use NeMo Curator Use NeMo Curator when: Preparing LLM training data from web scrapes (Common Crawl) Need fast deduplication (16× faster than CPU) Curating multi modal datasets (text, images, video, audio) Filtering low quality or toxic content Scaling data processing across GPU cluster Performance : 16× faster fuzzy deduplication (8TB RedPajama v2) 40% lower TCO vs CPU alternatives Near linear scaling across GPU nodes Use alternatives instead : datatrove : CPU based, open source data processing dolma : Allen AI's data toolkit Ray Data : General ML data processing (no curation focus) Quick start Installation Basic text curation pipeline Data curation pipeline Stage 1: Quality filtering Stage 2: Deduplication Exact deduplication : Fuzzy deduplication (16× faster on GPU): Semantic deduplication : Stage 3: PII redaction Stage 4: Classifier filtering GPU acceleration GPU vs CPU performance Operation CPU (16 cores) GPU (A100) Speedup Fuzzy dedup (8TB) 120 hours 7.5 hours 16× Exact dedup (1TB) 8 hours 0.5 hours 16× Quality filtering 2 hours 0.2 hours 10× Multi GPU scaling Multi modal curation Image curation Video curation Audio curation Common patterns Web scrape curation (Common Crawl) Distributed processing Performance benchmarks Fuzzy deduplication (8TB RedPajama v2) CPU (256 cores) : 120 hours GPU (8× A100) : 7.5 hours Speedup : 16× Exact deduplication (1TB) CPU (64 cores) : 8 hours GPU (4× A100) : 0.5 hours Speedup : 16× Quality filtering (100GB) CPU (32 cores) : 2 hours GPU (2× A100) : 0.2 hours Speedup : 10× Cost comparison CPU based curation (AWS c5.18xlarge × 10): Cost: $3.60/hour × 10 = $36/hour Time for 8TB: 120 hours Total : $4,320 GPU based curation (AWS p4d.24xlarge × 2): Cost: $32.77/hour × 2 = $65.54/hour Time for 8TB: 7.5 hours Total : $491.55 Savings : 89% reduction ($3,828 saved) Supported data formats Input : Parquet, JSONL, CSV Output : Parquet (recommended), JSONL WebDataset : TAR archives for multi modal Use cases Production deployments : NVIDIA used NeMo Curator to prepare Nemotron 4 training data Open source datasets curated: RedPajama v2, The Pile References [Filtering Guide](references/filtering.md) 30+ quality filters, heuristics [Deduplication Guide](references/deduplication.md) Exact, fuzzy, semantic methods Resources GitHub : https://github.com/NVIDIA/NeMo Curator ⭐ 500+ Docs : https://docs.nvidia.com/nemo framework/user guide/latest/datacuration/ Version : 0.4.0+ License : Apache 2.0

nemo-curator