Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK la

SentencePiece Language Independent Tokenization Unsupervised tokenizer that works on raw text without language specific preprocessing. When to use SentencePiece Use SentencePiece when: Building multilingual models (no language specific rules) Working with CJK languages (Chinese, Japanese, Korean) Need reproducible tokenization (deterministic vocabulary) Want to train on raw text (no pre tokenization needed) Require lightweight deployment (6MB memory, 50k sentences/sec) Performance : Speed : 50,000 sentences/sec Memory : ~6MB for loaded model Languages : All (language independent) Use alternatives instead : HuggingFace Tokenizers : Faster training, more flexibility tiktoken : OpenAI models (GPT 3.5/4) BERT WordPiece : English centric tasks Quick start Installation Train model Training time : ~1 2 minutes for 100MB corpus Encode and decode Language independent design Whitespace as symbol (▁) Key principle : Treat text as raw Unicode, whitespace = ▁ (meta symbol) Tokenization algorithms BPE (Byte Pair Encoding) Used by : mBART Unigram (default) Used by : T5, ALBERT, XLNet Training configuration Essential parameters Character coverage Language Type Coverage Rationale English 0.9995 Most common chars CJK (Chinese) 1.0 All characters needed Multilingual 0.9995 Balance Encoding options Subword regularization Use case : Data augmentation for robustness. Common patterns T5 style training Integration with transformers Performance benchmarks Training speed Corpus BPE (16k) Unigram (8k) 100 MB 1 2 min 3 4 min 1 GB 10 15 min 30 40 min Tokenization speed SentencePiece : 50,000 sentences/sec HF Tokenizers : 200,000 sentences/sec (4× faster) Supported models T5 family : t5 base , t5 large (32k vocab, Unigram) ALBERT : albert base v2 (30k vocab, Unigram) XLNet : xlnet base cased (32k vocab, Unigram) mBART : facebook/mbart large 50 (250k vocab, BPE) References [Training Guide](references/training.md) Detailed options, corpus preparation [Algorithms](references/algorithms.md) BPE vs Unigram, subword regularization Resources GitHub : https://github.com/google/sentencepiece ⭐ 10,000+ Paper : https://arxiv.org/abs/1808.06226 (EMNLP 2018) Version : 0.2.0+