Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

SGLang High performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching. When to use SGLang Use SGLang when: Need structured outputs (JSON, regex, grammar) Building agents with repeated prefixes (system prompts, tools) Agentic workflows with function calling Multi turn conversations with shared context Need faster JSON decoding (3× vs standard) Use vLLM instead when: Simple text generation without structure Don't need prefix caching Want mature, widely tested production system Use TensorRT LLM instead when: Maximum single request latency (no batching needed) NVIDIA only deployment Need FP8/INT4 quantization on H100 Quick start Installation Launch server Basic inference Structured JSON output RadixAttention (Key Innovation) What it does : Automatically caches and reuses common prefixes across requests. Performance : 5× faster for agentic workloads with shared system prompts 10× faster for few shot prompting with repeated examples Zero configuration works automatically How it works : 1. Builds radix tree of all processed tokens 2. Automatically detects shared prefixes 3. Reuses KV cache for matching prefixes 4. Only computes new tokens Example (Agent with system prompt): Structured generation patterns JSON with schema Regex constrained generation Grammar based generation python\n" EBNF grammar for Python python grammar = """ ?start: function def function def: "def" NAME "(" [parameters] "):" suite parameters: parameter ("," parameter) parameter: NAME suite: simple stmt NEWLINE INDENT stmt+ DEDENT """ s += sgl.gen("code", max tokens=200, grammar=python grammar) s += "\n Agent workflows with function calling Performance benchmarks RadixAttention speedup Few shot prompting (10 examples in prompt): vLLM: 2.5 sec/request SGLang: 0.25 sec/request (10× faster) Throughput: 4× higher Agent workflows (1000 token system prompt): vLLM: 1.8 sec/request SGLang: 0.35 sec/request (5× faster) JSON decoding : Standard: 45 tok/s SGLang: 135 tok/s (3× faster) Throughput (Llama 3 8B, A100) Workload vLLM SGLang Speedup Simple generation 2500 tok/s 2800 tok/s 1.12× Few shot (10 examples) 500 tok/s 5000 tok/s 10× Agent (tool calls) 800 tok/s 4000 tok/s 5× JSON output 600 tok/s 2400 tok/s 4× Multi turn conversations Advanced features Speculative decoding Multi modal (vision models) Batching and parallel requests OpenAI compatible API Supported models Text models : Llama 2, Llama 3, Llama 3.1, Llama 3.2 Mistral, Mixtral Qwen, Qwen2, QwQ DeepSeek V2, DeepSeek V3 Gemma, Phi 3 Vision models : LLaVA, LLaVA OneVision Phi 3 Vision Qwen2 VL 100+ models from HuggingFace Hardware support NVIDIA : A100, H100, L4, T4 (CUDA 11.8+) AMD : MI300, MI250 (ROCm 6.0+) Intel : Xeon with GPU (coming soon) Apple : M1/M2/M3 via MPS (experimental) References [Structured Generation Guide](references/structured generation.md) JSON schemas, regex, grammars, validation [RadixAttention Deep Dive](references/radix attention.md) How it works, optimization, benchmarks [Production Deployment](references/deployment.md) Multi GPU, monitoring, autoscaling Resources GitHub : https://github.com/sgl project/sglang Docs : https://sgl project.github.io/ Paper : RadixAttention (arXiv:2312.07104) Discord : https://discord.gg/sglang