Large Language and Vision Assistant. Enables visual instruction tuning and image-based conversations. Combines CLIP vision encoder with Vicuna/LLaMA language models. Supports multi-turn image chat, visual question answering, and instruction following. Use for vision-language chatbots or image unders

LLaVA Large Language and Vision Assistant Open source vision language model for conversational image understanding. When to use LLaVA Use when: Building vision language chatbots Visual question answering (VQA) Image description and captioning Multi turn image conversations Visual instruction following Document understanding with images Metrics : 23,000+ GitHub stars GPT 4V level capabilities (targeted) Apache 2.0 License Multiple model sizes (7B 34B params) Use alternatives instead : GPT 4V : Highest quality, API based CLIP : Simple zero shot classification BLIP 2 : Better for captioning only Flamingo : Research, not open source Quick start Installation Basic usage Available models Model Parameters VRAM Quality LLaVA v1.5 7B 7B ~14 GB Good LLaVA v1.5 13B 13B ~28 GB Better LLaVA v1.6 34B 34B ~70 GB Best CLI usage Web UI (Gradio) Multi turn conversations Common tasks Image captioning Visual question answering Object detection (textual) Scene understanding Document understanding Training custom model Quantization (reduce VRAM) Best practices 1. Start with 7B model Good quality, manageable VRAM 2. Use 4 bit quantization Reduces VRAM significantly 3. GPU required CPU inference extremely slow 4. Clear prompts Specific questions get better answers 5. Multi turn conversations Maintain conversation context 6. Temperature 0.2 0.7 Balance creativity/consistency 7. max new tokens 512 1024 For detailed responses 8. Batch processing Process multiple images sequentially Performance Model VRAM (FP16) VRAM (4 bit) Speed (tokens/s) 7B ~14 GB ~4 GB ~20 13B ~28 GB ~8 GB ~12 34B ~70 GB ~18 GB ~5 On A100 GPU Benchmarks LLaVA achieves competitive scores on: VQAv2 : 78.5% GQA : 62.0% MM Vet : 35.4% MMBench : 64.3% Limitations 1. Hallucinations May describe things not in image 2. Spatial reasoning Struggles with precise locations 3. Small text Difficulty reading fine print 4. Object counting Imprecise for many objects 5. VRAM requirements Need powerful GPU 6. Inference speed Slower than CLIP Integration with frameworks LangChain Gradio App Resources GitHub : https://github.com/haotian liu/LLaVA ⭐ 23,000+ Paper : https://arxiv.org/abs/2304.08485 Demo : https://llava.hliu.cc Models : https://huggingface.co/liuhaotian License : Apache 2.0