LLM Caching: Prompt Caching, Semantic Caching and KV Cache Reuse

LLM calls are slow and priced per token, so caching pays off more than almost anywhere else in a system. There are four distinct kinds: exact response caching (same request, same answer), prompt caching (the provider or server reuses work on a repeated prompt prefix), semantic caching (return a stored answer for a differently worded but equivalent question), and KV cache reuse inside a serving engine. Each has different savings and different ways to go wrong.

The four layers

1. Exact response caching

Key on a hash of everything that affects the output: model and version, prompt template version, parameters, retrieved context ids, and the input. Safe and simple; useful for repeated, deterministic requests (classifying the same product title, summarising the same document, popular FAQ answers at temperature 0).

• Include versions in the key so a prompt or model change never serves stale answers.
• Never share entries across users when the response depends on user data or permissions.
• Set TTLs by how fast the underlying data changes; see cache invalidation.

2. Prompt caching (prefix caching)

Many requests share a long identical prefix: the system prompt, tool definitions, few-shot examples, a large document being questioned repeatedly. Providers and self-hosted engines can store the computed state for that prefix and reuse it, charging less for cached input tokens and cutting time to first token. It never returns a wrong answer, because the model still generates fresh output.

• Put stable content first and variable content last; a single changed character early in the prompt breaks the shared prefix.
• Don't put timestamps or request ids at the top of the system prompt.
• Caches expire after minutes of inactivity; benefits are largest for steady traffic on the same prefix.

3. Semantic caching

Embed each incoming question, search a vector index of previously answered questions, and if the nearest one is above a similarity threshold, return its stored answer. It catches paraphrases ("how do I reset my password" vs "forgot password, what now"), which exact caching can't.

Semantic caching can return confidently wrong answers: "cancel my order" and "don't cancel my order" embed closely; "price in the US" and "price in the UK" may too. Use high thresholds, scope the cache to safe, non-personal, non-time-sensitive intents, and evaluate false hits explicitly.

4. KV cache reuse in serving engines

Inside an inference server, each request's attention keys and values (the KV cache) live in GPU memory during generation. Engines with prefix caching keep KV blocks for common prefixes and share them across requests, the self-hosted equivalent of provider prompt caching. Memory is the constraint: cached prefixes compete with active requests for GPU memory. See LLM serving.

Estimating the savings

Support bot: 10M requests/day, prompt = 3,000-token stable prefix + 1,500 variable
Prompt caching (say 90% prefix hit rate, cached tokens billed at ~10% of normal):
  prefix tokens/day   = 10M × 3,000 = 30B
  billed equivalent   = 30B × (0.1 × 0.9 + 1.0 × 0.1) = 5.7B  → ~80% off the prefix
Semantic cache on FAQ-type intents (say 25% of traffic, 60% hit rate within it):
  model calls avoided = 10M × 0.25 × 0.6 = 1.5M/day (15%), each answered in ~30 ms
(Illustrative rates; check your provider's actual pricing and cache behaviour.)

Operating caches

• Track hit rates, latency saved, cost saved and (for semantic caches) sampled false-hit rate.
• Invalidate on prompt, model or source-document changes; versioned keys make this automatic.
• Keep cache contents subject to the same privacy rules as logs: PII redaction, retention limits, per-tenant isolation.