Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth, live translati

Gemini Live API Development Skill Overview The Live API enables low latency, real time voice and video interactions with Gemini over WebSockets. It processes continuous streams of audio, video, or text to deliver immediate, human like spoken responses and background reasoning. Key capabilities: Bidirectional audio streaming — real time mic to speaker conversations Background reasoning (extended thinking) — multi step background reasoning with spoken conversational fillers Live streaming transcription — real time speech to text with interim and finalized streams Video streaming — send camera/screen frames alongside audio Text input/output — send and receive text within a live session Audio transcriptions — get text transcripts of both input and output audio Voice Activity Detection (VAD) — automatic server VAD, client side Hybrid VAD, and manual Push to Talk Asynchronous function calling — non blocking tool execution while audio continues streaming Full session client content — inject and update conversation turns mid stream Session management — context compression, session resumption, GoAway signals Ephemeral tokens — secure client side authentication [!NOTE] The Live API connects directly via WebSockets . For WebRTC support or simplified integration, use a [partner integration]( partner integrations). Models Current Models (Use These) gemini 3.8 live — Default option for most low latency voice agent experiences and real time dialogue without reasoning delays. Supports interleaved reasoning, asynchronous function calling by default ( behavior: NON BLOCKING ), and full session client content updates. gemini 3.8 live extended thinking — High reasoning audio to audio model recommended when higher background reasoning is required during live interactions. Processes background reasoning and async tool calls ( behavior: NON BLOCKING required) while streaming continuous spoken conversational fillers; lifecycle managed via interaction status ( IN PROGRESS vs IDLE ). gemini 3.5 transcribe live — Real time streaming speech to text with interim hypotheses, finalized transcripts, smart formatting, and Hybrid VAD. gemini 3.5 live translate preview — Real time speech to speech streaming translation across 70+ languages. [!WARNING] Legacy Models ( gemini 3.1 flash live preview , gemini 2.5 flash native audio , gemini live 2.5 flash preview , gemini 2.0 flash live 001 ) : Read [ references/migration.md ](references/migration.md) for breaking protocol changes ( behavior: "NON BLOCKING" , thinking level , interaction status , send client content ). SDKs Python : google genai = 2.3.0 — pip install U google genai JavaScript/TypeScript : @google/genai = 2.3.0 — npm install @google/genai [!WARNING] Legacy SDKs google generativeai (Python) and @google/generative ai (JS) are deprecated . Never use them. Partner Integrations To streamline real time audio/video app development, use a third party integration supporting the Gemini Live API over WebRTC or WebSockets : [LiveKit](https://docs.livekit.io/agents/models/realtime/plugins/gemini/) — Use the Gemini Live API with LiveKit Agents. [Pipecat by Daily](https://docs.pipecat.ai/guides/features/gemini live) — Create a real time AI chatbot using Gemini Live and Pipecat. [Fishjam by Software Mansion](https://docs.fishjam.io/tutorials/gemini live integration) — Create live video and audio streaming applications with Fishjam. [Vision Agents by Stream](https://visionagents.ai/integrations/gemini) — Build real time voice and video AI applications with Vision Agents. [Voximplant](https://voximplant.com/products/gemini client) — Connect inbound and outbound calls to Live API with Voximplant. [Firebase AI SDK](https://firebase.google.com/docs/ai logic/live api?api=dev) — Get started with the Gemini Live API using Firebase AI Logic. Audio Formats Input : Raw PCM, little endian, 16 bit, mono. 16kHz native (will resample others). MIME type: audio/pcm;rate=16000 Output : Raw PCM, little endian, 16 bit, mono. 24kHz sample rate. [!IMPORTANT] Use send realtime input / sendRealtimeInput for all real time streaming user input (audio, video, and text ). On Gemini 3.8 models, send client content / sendClientContent is supported across the full session lifecycle with explicit roles ( user or model ) to inject conversation context ( turn complete=true unconditionally interrupts active generation). [!WARNING] Do not use media in sendRealtimeInput . Use the specific keys: audio for audio data, video for images/video frames, and text for text input. Quick Start Authentication Python JavaScript Connecting to the Live API Python JavaScript Sending Text Python JavaScript Sending Audio Python JavaScript Sending Video Python JavaScript Receiving Audio and Text [!IMPORTANT] A single server event can contain multiple content parts simultaneously (e.g., audio chunks and transcript). Always process all parts in each event to avoid missing content. Python JavaScript Background Reasoning (Extended Thinking) Use gemini 3.8 live extended thinking when your voice agent must evaluate complex data, plan multiple steps, or handle long running tools. The model speaks natural conversational fillers (e.g. "Checking flight options now..." ) while executing asynchronous tools in the background. Key requirements: Thinking config : Set thinking config=types.ThinkingConfig(thinking level="low") ( "minimal" "low" "medium" "high" ). Non blocking tools : All function declarations must set behavior="NON BLOCKING" . Synchronous blocking mode is not supported and returns an error. Lifecycle tracking ( interaction status ) : Do not rely on turn complete=True alone to detect turn completion. Monitor message.interaction status (Python) / message.interactionStatus (JS): "IN PROGRESS" : Server is reasoning, speaking conversational fillers, or waiting for async tool responses. "IDLE" : Server has completed all background reasoning and tool calls; session is ready for user input. See [ references/migration.md ](references/migration.md) and the [Thinking in Live API Guide](https://ai.google.dev/gemini api/docs/live api/thinking.md.txt) for complete Python and JavaScript implementation examples. Live Translation (Gemini Live Translate) The Live API supports real time, low latency streaming translation of speech (audio) across 70+ languages. For full details on options and capabilities, see the [Live Translate Guide](https://ai.google.dev/gemini api/docs/live api/live translate.md.txt). Model gemini 3.5 live translate preview — The recommended translation model for all Live Translate use cases. Configuration ( TranslationConfig ) To enable translation, specify a TranslationConfig object inside your live session setup: Python SDK : Configure the connection using translation config on LiveConnectConfig : Raw WebSockets : Place translationConfig inside generationConfig : Live Streaming Transcription (Gemini Live Transcribe) The Live API supports real time streaming speech to text over WebSockets with low latency interim hypotheses, finalized transcripts, and Hybrid VAD. For full details, see the [Live Transcription Guide](https://ai.google.dev/gemini api/docs/live api/live transcribe.md.txt) and [Colab Cookbook](https://colab.research.google.com/github/google gemini/cookbook/blob/main/quickstarts/Get started transcribe.ipynb). Model gemini 3.5 transcribe live Modes smart : cleans up filler words, resolves inline self corrections, and structures formatting. verbatim (default): exact word for word transcript. Python JavaScript Raw WebSockets Limitations Response modality — Only TEXT or AUDIO per session, not both. Native audio models output audio ( response modalities=["AUDIO"] ); enable output audio transcription if you need text transcripts. Audio only session — 15 min without compression Audio+video session — 2 min without compression Connection lifetime — ~10 min (use session resumption) Context window — 128k input tokens / 64k output tokens Code execution / URL context — Not supported Upgrading & Migration For step by step migration checklists and protocol deltas when upgrading from gemini 3.1 flash live preview , gemini 2.5 flash native audio , or gemini 2.0 flash live 001 to Gemini 3.8 Live or Gemini 3.8 Live Extended Thinking , read [ references/migration.md ](references/migration.md). Best Practices 1. Use headphones when testing mic audio to prevent echo/self interruption 2. Enable context window compression for sessions longer than 15 minutes 3. Implement session resumption to handle connection resets gracefully 4. Use ephemeral tokens for client side deployments — never expose API keys in browsers 5. Use send realtime input for real time user input (audio, video, text). Use send client content with explicit user / model roles to inject context turns mid stream 6. Send audioStreamEnd / audio stream end (Hybrid VAD) when the mic is paused or user finishes speaking 7. Clear audio playback queues on interruption signals ( interrupted: true ) 8. Process all parts in each server event — events can contain multiple content parts 9. Monitor interaction status ( IN PROGRESS vs IDLE ) when using gemini 3.8 live extended thinking rather than relying on turn complete alone Documentation Lookup When MCP is Installed (Preferred) If the search docs tool (from the Google MCP server) is available, use it as your only documentation source: 1. Call search docs with your query 2. Read the returned documentation 3. Trust MCP results as source of truth for API details — they are always up to date. [!IMPORTANT] When MCP tools are present, never fetch URLs manually. MCP provides up to date, indexed documentation that is more accurate and token efficient than URL fetching. When MCP is NOT Installed (Fallback Only) If no MCP documentation tools are available, fetch from the official docs index: llms.txt URL : https://ai.google.dev/gemini api/docs/llms.txt This index contains links to all documentation pages in .md.txt format. Use web fetch tools to: 1. Fetch llms.txt to discover available documentation pages 2. Fetch specific pages (e.g., https://ai.google.dev/gemini api/docs/live session.md.txt ) Key Documentation Pages [!IMPORTANT] Those are not all the documentation pages. Use the llms.txt index to discover available documentation pages [Live API Overview](https://ai.google.dev/gemini api/docs/live.md.txt) — getting started, raw WebSocket usage [Thinking in Live API](https://ai.google.dev/gemini api/docs/live api/thinking.md.txt) — background reasoning, conversational fillers, interaction status, non blocking tools [Model Card: Gemini 3.8 Live](https://ai.google.dev/gemini api/docs/models/gemini 3.8 live) — default low latency voice agent model & migration guide [Model Card: Gemini 3.8 Live Extended Thinking](https://ai.google.dev/gemini api/docs/models/gemini 3.8 live extended thinking) — high reasoning voice model & upgrading guide [Live Transcription](https://ai.google.dev/gemini api/docs/live api/live transcribe.md.txt) — real time speech to text, interim hypotheses, smart formatting, and Hybrid VAD [Live Translate](https://ai.google.dev/gemini api/docs/live api/live translate.md.txt) — configuration options and capabilities for translation [Live API Capabilities Guide](https://ai.google.dev/gemini api/docs/live guide.md.txt) — voice config, transcription config, VAD configuration, media resolution [Live API Tool Use](https://ai.google.dev/gemini api/do

gemini-live-api-dev