Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jac

Speculative Decoding: Accelerating LLM Inference When to Use This Skill Use Speculative Decoding when you need to: Speed up inference by 1.5 3.6× without quality loss Reduce latency for real time applications (chatbots, code generation) Optimize throughput for high volume serving Deploy efficiently on limited hardware Generate faster without changing model architecture Key Techniques : Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration) Papers : Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024) Installation Quick Start Basic Speculative Decoding (Draft Model) Medusa (Multiple Decoding Heads) Lookahead Decoding (Jacobi Iteration) Core Concepts 1. Speculative Decoding (Draft Model) Idea : Use small draft model to generate candidates, large target model to verify in parallel. Algorithm : 1. Draft model generates K tokens speculatively 2. Target model evaluates all K tokens in parallel (single forward pass) 3. Accept tokens where draft and target agree 4. Reject first disagreement, continue from there Performance : Speedup: 1.5 2× with good draft model Zero quality loss (mathematically equivalent to target model) Best when draft model is 5 10× smaller than target 2. Medusa (Multiple Decoding Heads) Source : arXiv 2401.10774 (2024) Innovation : Add multiple prediction heads to existing model, predict future tokens without separate draft model. Architecture : Training : Medusa 1 : Freeze base LLM, train only heads 2.2× speedup, lossless Medusa 2 : Fine tune base LLM + heads together 2.3 3.6× speedup, better quality Tree based Attention : Advantages : No separate draft model needed Minimal training (only heads) Compatible with any LLM 3. Lookahead Decoding (Jacobi Iteration) Source : ICML 2024 Core idea : Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration. Mathematical formulation : Two branches : 1. Lookahead Branch : Generate n grams in parallel Window size W: How many steps to look ahead N gram size N: How many past tokens to use 2. Verification Branch : Verify promising n grams Match n grams with generated tokens Accept if first token matches Performance : Speedup: 1.5 2.3× (up to 3.6× for code generation) No draft model or training needed Works out of the box with any model Method Comparison Method Speedup Training Needed Draft Model Quality Loss Draft Model Speculative 1.5 2× No Yes (external) None Medusa 2 3.6× Minimal (heads only) No (built in heads) None Lookahead 1.5 2.3× None No None Naive Batching 1.2 1.5× No No None Advanced Patterns Training Medusa Heads Hybrid: Speculative + Medusa Optimal Draft Model Selection Best Practices 1. Choose the Right Method 2. Hyperparameter Tuning Draft Model Speculative : Medusa : Lookahead : 3. Production Deployment Resources Medusa Paper : https://arxiv.org/abs/2401.10774 Medusa GitHub : https://github.com/FasterDecoding/Medusa Lookahead Decoding (ICML 2024) : https://lmsys.org/blog/2023 11 21 lookahead decoding/ Lookahead GitHub : https://github.com/hao ai lab/LookaheadDecoding Speculative Decoding Survey (ACL 2024) : https://aclanthology.org/2024.findings acl.456.pdf Comprehensive Survey : https://arxiv.org/abs/2401.07851 See Also references/draft model.md Draft model selection and training references/medusa.md Medusa architecture and training references/lookahead.md Lookahead decoding implementation details

speculative-decoding