Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-t

GPTQ (Generative Pre trained Transformer Quantization) Post training quantization method that compresses LLMs to 4 bit with minimal accuracy loss using group wise quantization. When to use GPTQ Use GPTQ when: Need to fit large models (70B+) on limited GPU memory Want 4× memory reduction with <2% accuracy loss Deploying on consumer GPUs (RTX 4090, 3090) Need faster inference (3 4× speedup vs FP16) Use AWQ instead when: Need slightly better accuracy (<1% loss) Have newer GPUs (Ampere, Ada) Want Marlin kernel support (2× faster on some GPUs) Use bitsandbytes instead when: Need simple integration with transformers Want 8 bit quantization (less compression, better quality) Don't need pre quantized model files Quick start Installation Load pre quantized model Quantize your own model Group wise quantization How GPTQ works : 1. Group weights : Divide each weight matrix into groups (typically 128 elements) 2. Quantize per group : Each group has its own scale/zero point 3. Minimize error : Uses Hessian information to minimize quantization error 4. Result : 4 bit weights with near FP16 accuracy Group size trade off : Group Size Model Size Accuracy Speed Recommendation 1 (per column) Smallest Best Slowest Research only 32 Smaller Better Slower High accuracy needed 128 Medium Good Fast Recommended default 256 Larger Lower Faster Speed critical 1024 Largest Lowest Fastest Not recommended Example : Quantization configurations Standard 4 bit (recommended) Performance : Memory: 4× reduction (70B model: 140GB → 35GB) Accuracy: ~1.5% perplexity increase Speed: 3 4× faster than FP16 High accuracy (3 bit with larger groups) Trade off : Memory: 5× reduction Accuracy: ~3% perplexity increase Speed: 5× faster (but less accurate) Maximum accuracy (4 bit with small groups) Trade off : Memory: 3.5× reduction (slightly larger) Accuracy: ~0.8% perplexity increase (best) Speed: 2 3× faster (kernel overhead) Kernel backends ExLlamaV2 (default, fastest) Performance : 1.5 2× faster than Triton Marlin (Ampere+ GPUs) Requirements : NVIDIA Ampere or newer (A100, H100, RTX 40xx) Compute capability ≥ 8.0 Triton (Linux only) Performance : 1.2 1.5× faster than CUDA backend Integration with transformers Direct transformers usage QLoRA fine tuning (GPTQ + LoRA) Performance benchmarks Memory reduction Model FP16 GPTQ 4 bit Reduction Llama 2 7B 14 GB 3.5 GB 4× Llama 2 13B 26 GB 6.5 GB 4× Llama 2 70B 140 GB 35 GB 4× Llama 3 405B 810 GB 203 GB 4× Enables : 70B on single A100 80GB (vs 2× A100 needed for FP16) 405B on 3× A100 80GB (vs 11× A100 needed for FP16) 13B on RTX 4090 24GB (vs OOM with FP16) Inference speed (Llama 2 7B, A100) Precision Tokens/sec vs FP16 FP16 25 tok/s 1× GPTQ 4 bit (CUDA) 85 tok/s 3.4× GPTQ 4 bit (ExLlama) 105 tok/s 4.2× GPTQ 4 bit (Marlin) 120 tok/s 4.8× Accuracy (perplexity on WikiText 2) Model FP16 GPTQ 4 bit (g=128) Degradation Llama 2 7B 5.47 5.55 +1.5% Llama 2 13B 4.88 4.95 +1.4% Llama 2 70B 3.32 3.38 +1.8% Excellent quality preservation less than 2% degradation! Common patterns Multi GPU deployment CPU offloading Batch inference Finding pre quantized models TheBloke on HuggingFace : https://huggingface.co/TheBloke 1000+ models in GPTQ format Multiple group sizes (32, 128) Both CUDA and Marlin formats Search : Download : Supported models LLaMA family : Llama 2, Llama 3, Code Llama Mistral : Mistral 7B, Mixtral 8x7B, 8x22B Qwen : Qwen, Qwen2, QwQ DeepSeek : V2, V3 Phi : Phi 2, Phi 3 Yi, Falcon, BLOOM, OPT 100+ models on HuggingFace References [Calibration Guide](references/calibration.md) Dataset selection, quantization process, quality optimization [Integration Guide](references/integration.md) Transformers, PEFT, vLLM, TensorRT LLM [Troubleshooting](references/troubleshooting.md) Common issues, performance optimization Resources GitHub : https://github.com/AutoGPTQ/AutoGPTQ Paper : GPTQ: Accurate Post Training Quantization (arXiv:2210.17323) Models : https://huggingface.co/models?library=gptq Discord : https://discord.gg/autogptq