prompt-caching
Caching strategies for LLM prompts including Anthropic prompt caching, response caching, and CAG (Cache Augmented Generation)
By sickn33 · 904 installs
npx skills add sickn33/agentic-awesome-skills --skill prompt-caching
Source repository · Upstream listing
Prompt Caching
Caching strategies for LLM prompts including Anthropic prompt caching, response caching, and CAG (Cache Augmented Generation)
Capabilities
prompt cache
response cache
kv cache
cag patterns
cache invalidation
Prerequisites
Knowledge: Caching fundamentals, LLM API usage, Hash functions
Skills recommended: context window management
Scope
Does not cover: CDN caching, Database query caching, Static asset caching
Boundaries: Focus is LLM specific caching, Covers prompt and response caching
Ecosystem
Primary tools
Anthropic Prompt Caching Native prompt caching in Claude API
Redis In memory cache for responses (ioredis on servers; @upstash/redis over HTTP on serverless, see upstash redis )
OpenAI Caching Automatic caching in OpenAI API
Patterns
Anthropic Prompt Caching
Use Claude's native prompt caching for repeated prefixes
When to use : Using Claude API with stable system prompts or context
import Anthropic from '@anthropic ai/sdk';
const client = new Anthropic();
// Cache the stable parts of your prompt
async function queryWithCaching(userQuery: string) {
const response = await client.messages.create({
model: "claude sonnet 4 20250514",
max tokens: 1024,
system: [
{
type: "text",
text: LONG SYSTEM PROMPT, // Your detailed instructions
cache control: { type: "ephemeral" } // Cache this!
},
{
type: "text",
text: KNOWLEDGE BASE, // Large static context
cache control: { type: "ephemeral" }
}
],
messages: [
{ role: "user", content: userQuery } // Dynamic part
]
});
// Check cache usage
console.log( Cache read: ${response.usage.cache read input tokens} );
console.log( Cache write: ${response.usage.cache creation input tokens} );
return response;
}
// Cost savings: 90% reduction on cached tokens
// Latency savings: Up to 2x faster
Response Caching
Cache full LLM responses for identical or similar queries
When to use : Same queries asked repeatedly
import { createHash } from 'crypto';
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS URL);
// Serverless/edge alternative without a persistent connection:
// import { Redis } from '@upstash/redis'; const redis = Redis.fromEnv(); // then use redis.set(key, value, { ex: ttl })
class ResponseCache {
private ttl = 3600; // 1 hour default
// Exact match caching
async getCached(prompt: string): Promise<string null {
const key = this.hashPrompt(prompt);
return await redis.get( response:${key} );
}
async setCached(prompt: string, response: string): Promise<void {
const key = this.hashPrompt(prompt);
await redis.set( response:${key} , response, 'EX', this.ttl);
}
private hashPrompt(prompt: string): string {
return createHash('sha256').update(prompt).digest('hex');
}
// Semantic similarity caching
async getSemanticallySimilar(
prompt: string,
threshold: number = 0.95
): Promise<string null {
const embedding = await embed(prompt);
const similar = await this.vectorCache.search(embedding, 1);
if (similar.length && similar[0].similarity threshold) {
return await redis.get( response:${similar[0].id} );
}
return null;
}
// Temperature aware caching
async getCachedWithParams(
prompt: string,
params: { temperature: number; model: string }
): Promise<string null {
// Only cache low temperature responses
if (params.temperature 0.5) return null;
const key = this.hashPrompt(
${prompt} ${params.model} ${params.temperature}
);
return await redis.get( response:${key} );
}
}
Cache Augmented Generation (CAG)
Pre cache documents in prompt instead of RAG retrieval
When to use : Document corpus is stable and fits in context
// CAG: Pre compute document context, cache in prompt
// Better than RAG when:
// Documents are stable
// Total fits in context window
// Latency is critical
class CAGSystem {
private cachedContext: string null = null;
private lastUpdate: number = 0;
async buildCachedContext(documents: Document[]): Promise<void {
// Pre process and format documents
const formatted = documents.map(d =
${d.title}\n${d.content}
).join('\n\n');
// Store with timestamp
this.cachedContext = formatted;
this.lastUpdate = Date.now();
}
async query(userQuery: string): Promise<string {
// Use cached context directly in prompt
const response = await client.messages.create({
model: "claude sonnet 4 20250514",
max tokens: 1024,
system: [
{
type: "text",
text: "You are a helpful assistant with access to the following documentation.",
cache control: { type: "ephemeral" }
},
{
type: "text",
text: this.cachedContext!, // Pre cached docs
cache control: { type: "ephemeral" }
}
],
messages: [{ role: "user", content: userQuery }]
});
return response.content[0].text;
}
// Periodic refresh
async refreshIfNeeded(documents: Document[]): Promise<void {
const stale = Date.now() this.lastUpdate 3600000; // 1 hour
if (stale) {
await this.buildCachedContext(documents);
}
}
}
// CAG vs RAG decision matrix:
// Factor CAG Better RAG Better
//
// Corpus size < 100K tokens 100K tokens
// Update frequency Low High
// Latency needs Critical Flexible
// Query specificity General Specific
Sharp Edges
Cache miss causes latency spike with additional overhead
Severity: HIGH
Situation: Slow response when cache miss, slower than no caching
Symptoms:
Slow responses on cache miss
Cache hit rate below 50%
Higher latency than uncached
Why this breaks:
Cache check adds latency.
Cache write adds more latency.
Miss + overhead no caching.
Recommended fix:
// Optimize for cache misses, not just hits
class OptimizedCache {
async queryWithCache(prompt: string): Promise<string {
const cacheKey = this.hash(prompt);
// Non blocking cache check
const cachedPromise = this.cache.get(cacheKey);
const llmPromise = this.queryLLM(prompt);
// Race: use cache if available before LLM returns
const cached = await Promise.race([
cachedPromise,
sleep(50).then(() = null) // 50ms cache timeout
]);
if (cached) {
// Cancel LLM request if possible
return cached;
}
// Cache miss: continue with LLM
const response = await llmPromise;
// Async cache write (don't block response)
this.cache.set(cacheKey, response).catch(console.error);
return response;
}
}
// Alternative: Probabilistic caching
// Only cache if query matches known high frequency patterns
class SelectiveCache {
private patterns: Map<string, number = new Map();
shouldCache(prompt: string): boolean {
const pattern = this.extractPattern(prompt);
const frequency = this.patterns.get(pattern) 0;
// Only cache high frequency patterns
return frequency 10;
}
recordQuery(prompt: string): void {
const pattern = this.extractPattern(prompt);
this.patterns.set(pattern, (this.patterns.get(pattern) 0) + 1);
}
}
Cached responses become incorrect over time
Severity: HIGH
Situation: Users get outdated or wrong information from cache
Symptoms:
Users report wrong information
Answers don't match current data
Complaints about outdated responses
Why this breaks:
Source data changed.
No cache invalidation.
Long TTLs for dynamic data.
Recommended fix:
// Implement proper cache invalidation
class InvalidatingCache {
// Version based invalidation
private cacheVersion = 1;
getCacheKey(prompt: string): string {
return v${this.cacheVersion}:${this.hash(prompt)} ;
}
invalidateAll(): void {
this.cacheVersion++;
// Old keys automatically become orphaned
}
// Content hash invalidation
async setWithContentHash(
key: string,
response: string,
sourceContent: string
): Promise<void {
const contentHash = this.hash(sourceContent);
await this.cache.set(key, {
response,
contentHash,
timestamp: Date.now()
});
}
async getIfValid(
key: string,
currentSourceContent: string
): Promise<string null {
const cached = await this.cache.get(key);
if (!cached) return null;
// Check if source content changed
const currentHash = this.hash(currentSourceContent);
if (cached.contentHash !== currentHash) {
await this.cache.delete(key);
return null;
}
return cached.response;
}
// Event based invalidation
onSourceUpdate(sourceId: string): void {
// Invalidate all caches that used this source
this.invalidateByTag( source:${sourceId} );
}
}
Prompt caching doesn't work due to prefix changes
Severity: MEDIUM
Situation: Cache misses despite similar prompts
Symptoms:
Cache hit rate lower than expected
Cache creation tokens high, read low
Similar prompts not hitting cache
Why this breaks:
Anthropic caching requires exact prefix match.
Timestamps or dynamic content in prefix.
Different message order.
Recommended fix:
// Structure prompts for optimal caching
class CacheOptimizedPrompts {
// WRONG: Dynamic content in cached prefix
buildPromptBad(query: string): SystemMessage[] {
return [
{
type: "text",
text: You are helpful. Current time: ${new Date()} , // BREAKS CACHE!
cache control: { type: "ephemeral" }
}
];
}
// RIGHT: Static prefix, dynamic at end
buildPromptGood(query: string): SystemMessage[] {
return [
{
type: "text",
text: STATIC SYSTEM PROMPT, // Never changes
cache control: { type: "ephemeral" }
},
{
type: "text",
text: STATIC KNOWLEDGE BASE, // Rarely changes
cache control: { type: "ephemeral" }
}
// Dynamic content goes in messages, NOT system
];
}
// Prefix ordering matters
buildWithConsistentOrder(components: string[]): SystemMessage[] {
// Sort components for consistent ordering
const sorted = [...components].sort();
return sorted.map((c, i) = ({
type: "text",
text: c,
cache control: i === sorted.length 1
? { type: "ephemeral" }
: undefined // Only cache the full prefix
}));
}
}
Validation Checks
Caching High Temperature Responses
Severity: WARNING
Message: Caching with high temperature. Responses are non deterministic.
Fix action: Only cache responses with temperature <= 0.5
Cache Without TTL
Severity: WARNING
Message: Cache without TTL. May serve stale data indefinitely.
Fix action: Set appropriate TTL based on data freshness requirements
Dynamic Content in Cached Prefix
Severity: WARNING
Message: Dynamic content in cached prefix. Will cause cache misses.
Fix action: Move dynamic content outside of cache control blocks
No Ca