Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipe

Ray Data Scalable ML Data Processing Distributed data processing library for ML and AI workloads. When to use Ray Data Use Ray Data when: Processing large datasets ( 100GB) for ML training Need distributed data preprocessing across cluster Building batch inference pipelines Loading multi modal data (images, audio, video) Scaling data processing from laptop to cluster Key features : Streaming execution : Process data larger than memory GPU support : Accelerate transforms with GPUs Framework integration : PyTorch, TensorFlow, HuggingFace Multi modal : Images, Parquet, CSV, JSON, audio, video Use alternatives instead : Pandas : Small data (<1GB) on single machine Dask : Tabular data, SQL like operations Spark : Enterprise ETL, SQL queries Quick start Installation Load and transform data Integration with Ray Train Reading data From cloud storage From Python objects Transformations Map batches (vectorized) Row transformations Filter Group by and aggregate GPU accelerated transforms Writing data Performance optimization Repartition Batch size tuning Streaming execution Common patterns Batch inference Data preprocessing pipeline Integration with ML frameworks PyTorch TensorFlow Supported data formats Format Read Write Use Case Parquet ✅ ✅ ML data (recommended) CSV ✅ ✅ Tabular data JSON ✅ ✅ Semi structured Images ✅ ❌ Computer vision NumPy ✅ ✅ Arrays Pandas ✅ ❌ DataFrames Performance benchmarks Scaling (processing 100GB data): 1 node (16 cores): ~30 minutes 4 nodes (64 cores): ~8 minutes 16 nodes (256 cores): ~2 minutes GPU acceleration (image preprocessing): CPU only: 1,000 images/sec 1 GPU: 5,000 images/sec 4 GPUs: 18,000 images/sec Use cases Production deployments : Pinterest : Last mile data processing for model training ByteDance : Scaling offline inference with multi modal LLMs Spotify : ML platform for batch inference References [Transformations Guide](references/transformations.md) Map, filter, groupby operations [Integration Guide](references/integration.md) Ray Train, PyTorch, TensorFlow Resources Docs : https://docs.ray.io/en/latest/data/data.html GitHub : https://github.com/ray project/ray ⭐ 36,000+ Version : Ray 2.40.0+ Examples : https://docs.ray.io/en/latest/data/examples/overview.html

ray-data