voice-agents
Voice agents represent the frontier of AI interaction - humans speaking naturally with AI systems.
By sickn33 · 924 installs
npx skills add sickn33/agentic-awesome-skills --skill voice-agents
Source repository · Upstream listing
Voice Agents
Voice agents represent the frontier of AI interaction humans speaking
naturally with AI systems. The challenge isn't just speech recognition
and synthesis, it's achieving natural conversation flow with sub 800ms
latency while handling interruptions, background noise, and emotional
nuance.
This skill covers two architectures: speech to speech (OpenAI Realtime API,
lowest latency, most natural) and pipeline (STT→LLM→TTS, more control,
easier to debug). Key insight: latency is the constraint. Humans expect
responses in 500ms. Every millisecond matters.
84% of organizations are increasing voice AI budgets in 2025. This is the
year voice agents go mainstream.
Detailed Guide
Read [the detailed guide](references/detailed guide.md) before executing this skill. It retains the complete procedure and reference material. Treat its safety, prerequisites, and validation requirements as mandatory. For focused work, load the relevant sections; for end to end work, read the guide completely.
Production Pipeline Example
"""
import { Deepgram } from '@deepgram/sdk';
import { ElevenLabsClient } from 'elevenlabs';
import OpenAI from 'openai';
// Initialize clients
const deepgram = new Deepgram(process.env.DEEPGRAM API KEY);
const elevenlabs = new ElevenLabsClient();
const openai = new OpenAI();
async function processVoiceInput(audioStream) {
// 1. Speech to Text (Deepgram Nova 3)
const transcription = await deepgram.transcription.live({
model: 'nova 3',
punctuate: true,
endpointing: 300, // ms of silence before end
});
transcription.on('transcript', async (data) = {
if (data.is final && data.speech final) {
const userText = data.channel.alternatives[0].transcript;
console.log('User:', userText);
// 2. LLM Processing
const completion = await openai.chat.completions.create({
model: 'gpt 4o mini',
messages: [
{ role: 'system', content: 'You are a concise voice assistant.' },
{ role: 'user', content: userText }
],
max tokens: 150, // Keep responses short for voice
});
const agentText = completion.choices[0].message.content;
console.log('Agent:', agentText);
// 3. Text to Speech (ElevenLabs)
const audioStream = await elevenlabs.textToSpeech.stream({
voice id: 'voice id here',
text: agentText,
model id: 'eleven flash v2 5', // Lowest latency
});
// Stream to user
playAudioStream(audioStream);
}
});
// Pipe audio to transcription
audioStream.pipe(transcription);
}
"""
Optimization Tips:
Start TTS while LLM still generating (streaming)
Pre compute first response segment during user speech
Use Flash/turbo models for latency
Voice Activity Detection Pattern
Detect when user starts/stops speaking
When to use : All voice agents need VAD for turn taking
VOICE ACTIVITY DETECTION (VAD):
"""
VAD Types:
1. Energy based: Simple, fast, noise sensitive
2. Model based: Silero VAD, more accurate
3. Semantic VAD: Understands meaning, best for conversation
"""
When to Use
User mentions or implies: voice agent
User mentions or implies: speech to text
User mentions or implies: text to speech
User mentions or implies: whisper
User mentions or implies: elevenlabs
User mentions or implies: deepgram
User mentions or implies: realtime api
User mentions or implies: voice assistant
User mentions or implies: voice ai
User mentions or implies: conversational ai
User mentions or implies: tts
User mentions or implies: stt
User mentions or implies: asr
Limitations
Use this skill only when the task clearly matches the scope described above.
Do not treat the output as a substitute for environment specific validation, testing, or expert review.
Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.