Caching LLM Responses: Cutting Cost Without Cutting Quality

A support assistant we inherited last year was burning roughly $840 a day in model spend on 38,000 requests. Nobody had looked at the prompt in four months. When we logged the actual traffic, 31% of requests were character-for-character duplicates of something answered in the previous hour, and every single request was re-sending the same 4,200 token system prompt and product summary at full price.

Three days of work took that to about $290 a day. No model downgrade, no shorter answers, no measurable change in the eval suite. Most of the win came from two mechanisms that get lumped together under llm caching but behave nothing alike: provider-side prompt caching, and an exact-match response cache in front of the API.

The third mechanism, semantic caching, is the one every listicle leads with. It’s also the one most likely to quietly hand a customer the wrong answer. We’ll get to why.

Three caches, and only one of them is risky

Conflating these three is the root of most bad advice on the topic. They sit at different layers, have different failure modes and different cost curves.

1. Provider prompt caching. The provider keeps the attention KV state for a prompt prefix warm on their side and charges you a fraction of the normal input rate to reuse it. The model still runs. Output is freshly generated every time. Quality risk: zero. There is no scenario where prompt caching changes the answer you’d otherwise get.
2. Exact-match response caching. You hash the full request and store the completion in Redis or similar. A hit skips the API entirely. Cost on hit: zero plus a sub-millisecond Redis round trip. Quality risk: staleness only, which you control with TTLs and invalidation.
3. Semantic caching. You embed the incoming query, search a vector index of previous queries, and if something is “close enough” you return its stored answer. Cost on hit: one embedding call. Quality risk: substantial and non-obvious.

Deploy them in that order. We’ve never seen a project where skipping straight to semantic caching was the right call. Layers one and two typically get you 60% to 70% of the available savings with essentially no downside.

Prompt caching cuts the input bill, not the output bill

The mechanics differ by provider but the shape is the same. Anthropic charges 1.25x base input rate to write a cache entry with a 5 minute TTL (2x for the 1 hour option) and 0.1x to read it, with a minimum cacheable prefix of 1,024 tokens on most models and up to four breakpoints. OpenAI does automatic prefix caching above 1,024 tokens with no write premium, and the read discount varies by model generation, roughly 50% on the GPT-4o era and up to 90% on newer ones. Gemini runs implicit caching by default and offers explicit context caching with a storage charge per token-hour, which only makes sense for genuinely large contexts you query repeatedly.

The hard part isn’t enabling it. It’s prompt architecture. Caching works on exact prefix matches, so one variable token near the top destroys everything below it.

The pattern that keeps failing: someone puts Current date: 2026-02-14T09:31:02Z in the first line of the system prompt. Every request now has a unique prefix. Cache hit rate: 0%. The fix is to round that timestamp to the day, or better, move it to the end of the user turn where it belongs.

const res = await anthropic.messages.create({
  model: 'claude-sonnet-4-5',
  max_tokens: 1024,
  system: [
    // Stable blocks first, longest and least volatile at the top.
    { type: 'text', text: TOOL_INSTRUCTIONS },      // ~900 tokens, changes on deploy
    { type: 'text', text: PRODUCT_CATALOGUE,        // ~3,300 tokens, changes weekly
      cache_control: { type: 'ephemeral' } }        // breakpoint AFTER everything stable
  ],
  messages: [
    // Volatile content lives here, below the breakpoint.
    { role: 'user', content: Today is ${today}.\n\n${retrievedChunks}\n\n${question} }
  ]
});

// Log these on every call. If cachereadinput_tokens is 0, your prefix is broken.
log.info({
  read: res.usage.cachereadinput_tokens,
  written: res.usage.cachecreationinput_tokens
});

Run the arithmetic on your own traffic before you celebrate. At Sonnet-class list pricing ($3 per million input, $15 per million output), that client’s 4,200 token static prefix across 38,000 daily requests cost $479 a day uncached. Served from cache at 0.1x, plus cache writes refreshed every five minutes, it came to about $52. The remaining 1,600 variable tokens and all 320 output tokens per request were untouched. That’s exactly why prompt caching alone only halved the bill.

When prompt caching does not pay

Short prompts under the minimum threshold get nothing. High-cardinality prefixes (per-user system prompts, personalised instructions injected at the top) get nothing. On Anthropic’s write premium, a prefix you touch once every twenty minutes with a 5 minute TTL costs you 1.25x repeatedly and returns nothing. Traffic has to be dense enough to keep entries warm.

Exact-match response caching: the key is the whole problem

This is the layer that actually eliminates cost, and it’s also where we see the ugliest bugs. Not because the caching is hard, but because people build the key carelessly.

import { createHash } from 'node:crypto';

const CACHE_VERSION = 'v4'; // bump on prompt change to invalidate every entry at once

const sha = (s) => createHash('sha256').update(s).digest('hex');

export function cacheKey({ model, system, question, params, tenantId, toolSchemaHash }) {
  const normalised = question
    .replace(/\r\n/g, '\n')
    .replace(/[ \t]+/g, ' ')
    .trim();                 // whitespace only. Lowercasing breaks code, SKUs and proper nouns.

  return llm:${CACHE_VERSION}: + sha(JSON.stringify({
    model,                   // claude-sonnet-4-5 and claude-haiku-4-5 must never share an entry
    temperature: params.temperature,
    topp: params.topp,
    maxtokens: params.maxtokens,
    responseformat: params.responseformat ?? null,
    system: sha(system),
    toolSchemaHash,          // changing a tool's JSON schema changes the answer
    tenantId,                // omit this and you leak answers across customers
    normalised
  }));
}

Two opinions worth arguing with. First: aggressive normalisation is a trap. Stripping punctuation turns “Let’s eat, Grandma” into something else, and lowercasing a prompt containing a case-sensitive API key or a SKU gives you a hit on a genuinely different question. Normalise whitespace and line endings. Stop there.

Second: cache the structured result, not the prose. If the model returns JSON you then render, store the parsed and validated object, keyed by schema version. You get free re-rendering when the template changes, and you can inspect what you cached. The same argument we make about versioning webhook payloads applies here: a cached artefact is a contract, and contracts need version numbers.

On hit rates: anything over 25% on conversational traffic is good. Single-turn classification, moderation, tagging and extraction jobs routinely hit 60% to 85% because the input distribution is narrow. Long multi-turn chat hits almost nothing at exact match, because the conversation history makes every request unique. Accept that and spend your effort on prompt caching for those paths instead.

Semantic caching is where quality quietly dies

The pitch is seductive: embed the query, find a similar past query, return its answer. In practice, cosine similarity measures topical overlap, not semantic equivalence, and the gap between those two concepts is exactly where your support team starts getting angry tickets.

Try these pairs against any current embedding model:

• “Can I cancel after 30 days?” versus “Can I cancel before 30 days?”
• “Is the Pro plan included in the trial?” versus “Is the Pro plan excluded from the trial?”
• “Ship to Ireland” versus “Ship to Iceland”
• “What’s my invoice total?” versus “What was my invoice total last year?”

Every pair lands high, often above 0.94. Negation barely moves an embedding. Numbers and dates move it less than you’d expect. A threshold alone cannot save you, so the gate needs explicit rules on top:

const POLARITY = /\b(not|never|no longer|except|without|cancel|refund|exclude|instead)\b/i;
const nums = (s) => (s.match(/\d+(?:[.,]\d+)?/g) ?? []).join('|');

function eligible(query, hit) {
  if (hit.score  6  60  60 * 1000) return false;      // short TTL: policy text changes
  if (hit.confidence === 'low') return false;             // never promote a hedged answer
  return true;
}

Our position: semantic caching belongs on FAQ-shaped, read-only, non-personalised, non-transactional queries. Documentation lookups. Product questions. “How do I reset my password.” It does not belong anywhere near account data, pricing quotes, order status, medical or legal content, or anything that writes to a database. On client work we gate it behind an intent classifier and whitelist the intents, rather than letting it see all traffic.

TTLs, invalidation and a staleness budget

Pick a staleness budget per content class and write it down, because “how long is too old” is a product decision, not an engineering one. The split that has held up across projects:

• Immutable derivations (summarise this uploaded PDF, extract entities from this fixed string): cache indefinitely, keyed by content hash. There is no staleness.
• Documentation and policy answers: 6 to 24 hours, with event-based purge wired into your CMS publish hook.
• Anything touching live data: don’t cache the answer. Cache the retrieval, or the embedding, or nothing.

Event-based invalidation beats short TTLs. Tag entries by the source documents that fed them, then purge by tag when a document changes. If you’ve built tag-based purging for a custom REST endpoint with proper cache headers, it’s the same mechanism, just with a vector or hash key instead of a URL.

One operational detail people miss: cache the fact that a request failed validation, but never cache a truncated response. If stop_reason is a length cap, or a tool call came back malformed, or your JSON schema validation failed, write nothing. We’ve debugged a production cache serving a half-finished sentence to thousands of users for two days because the write happened before validation.

Proving you did not cut quality

Cost savings are trivially measurable. Quality regressions are not, which is why most teams ship a cache and then have no idea whether it hurt. Three things make this tractable.

A frozen eval set. Between 100 and 300 real queries with graded reference answers, run through the full stack with the cache enabled and disabled. If the cached path scores within noise of the uncached path, you’re fine. If it doesn’t, your similarity threshold or TTL is wrong.

Shadow sampling. Route 1% to 5% of cache-eligible traffic past the cache, call the model anyway, and diff the two answers asynchronously. A rising divergence rate is your early warning that the underlying data moved and your TTLs are too long. This costs you a few percent of the savings, and it’s the best money in the whole system.

Per-layer attribution. Log which layer served every response: exact, semantic, promptcachehit, cold. Then track thumbs-down rate, escalation rate and follow-up rate per layer. On one deployment, semantic hits showed a 2.4x higher follow-up question rate than cold responses, which told us the answers were technically related but not actually responsive. We cut the eligible intents from nine to three.

This kind of instrumented, measure-then-decide discipline is the same reason we’re picky about which AI tools earn a place in the workflow. Anything that can’t be measured gets removed eventually.

The default stack we ship

For a typical client assistant or content pipeline, here’s the configuration we start from and tune down:

1. Restructure the prompt so the stable prefix is first and over 1,024 tokens, with volatile content below the breakpoint. Verify with cachereadinput_tokens in production logs, not locally.
2. Redis exact-match cache with a fully specified key, 1 hour TTL for conversational paths, content-hash keys with no expiry for deterministic transforms. Writes gated behind schema validation.
3. Cache the embedding calls too. They’re cheap individually and enormous in aggregate on any RAG pipeline, and embeddings of fixed text never go stale.
4. Semantic cache off by default. Enabled per intent, threshold 0.95 plus polarity and numeric guardrails, 6 hour TTL, logged separately.
5. 5% shadow sample, weekly eval run in CI against the frozen set.

That combination is what took the $840 a day case to roughly $290 without touching the model or the output length. Roughly half from prompt caching, most of the rest from exact-match hits, and almost nothing from semantic, which we eventually limited to three intents.