Choosing an LLM API: Cost, Latency and Lock-in

A client came to us in March with a $9,400 monthly bill for a support summarisation feature that handled about 40,000 tickets. The feature worked fine. The problem was that every single request re-sent a 6,000 token system prompt containing their entire product taxonomy, uncached, to a frontier model, then asked for a free-form summary that averaged 400 tokens when 60 would have done. Two changes, prompt caching and a hard output cap, took it to roughly $1,100. No model change. No quality drop that their evals could detect.

That’s the shape of most LLM cost problems. It’s rarely “we picked the wrong provider” and almost always “we’re paying for tokens we never needed to send or generate”. Provider choice does matter, but it matters in ways spec-sheet comparisons miss: rate limit headroom under burst, tail latency at p99, and how deeply your prompts have grown into one vendor’s quirks.

This is the llm api comparison we actually run internally before committing a client project to a provider. Less about benchmark scores, more about the three things that bite you in month four.

The three numbers that decide your bill

Forget the pricing page for a second. Your monthly cost is: requests per month, multiplied by (uncached input tokens x input rate) + (cached input tokens x cached rate) + (output tokens x output rate). That’s it. Everything else is noise.

What makes this non-obvious is the ratio between those rates. Output tokens typically cost three to five times input tokens on the same model. Cached input reads cost around a tenth of fresh input. So a request with 8,000 cached input tokens and 100 output tokens can be cheaper than one with 500 fresh input tokens and 600 output tokens, even though it “looks” sixteen times bigger.

Build the estimator before you build the feature:

// Illustrative tier rates in USD per 1M tokens. Always verify against the
// live pricing page: these move downward every few months.
const TIERS = {
  small:    { in: 0.15, cached: 0.015, out: 0.60 },
  mid:      { in: 3.00, cached: 0.30,  out: 15.00 },
  frontier: { in: 15.00, cached: 1.50, out: 75.00 }
};

function monthlyCost({ tier, requests, freshIn, cachedIn, out }) {
  const r = TIERS[tier];
  const per = (freshIn  r.in + cachedIn  r.cached + out * r.out) / 1e6;
  return { perRequest: per, monthly: per * requests };
}

// The support summariser, before and after.
console.log(monthlyCost({ tier: 'frontier', requests: 40000,
  freshIn: 6400, cachedIn: 0, out: 400 }));   // ~$5.0k/mo
console.log(monthlyCost({ tier: 'mid', requests: 40000,
  freshIn: 400, cachedIn: 6000, out: 120 })); // ~$0.16k/mo

Run that with your real numbers before you write a line of prompt. If the answer is “we cannot afford this at any provider”, the feature needs redesigning, not shopping around.

OpenAI vs Anthropic API: what actually differs

Both are HTTPS endpoints that take messages and return tokens. The differences that cost you time are structural, not philosophical.

Request shape

Anthropic’s Messages API puts the system prompt at the top level as its own parameter, and max_tokens is required, not optional. OpenAI historically treated system as a message role, and its newer Responses API introduced input and instructions alongside the older Chat Completions shape. If you wrote against Chat Completions in 2023, you have a migration ahead of you at some point. Worth knowing before you argue that OpenAI is the “stable” choice.

Caching

Anthropic’s caching is explicit. You mark a content block with cache_control and you control what gets cached. Writes cost 1.25x the base input rate, reads cost 0.1x, and the default TTL is five minutes with a one hour option at a higher write price. OpenAI’s caching is automatic on prefixes above 1024 tokens with no annotation required. Explicit caching gives you more control and more ways to get it wrong. Automatic caching is easier and silently does nothing if you put a timestamp at the top of your prompt.

// Anthropic: cache the stable taxonomy, leave the ticket fresh.
body: JSON.stringify({
  model: 'claude-sonnet-4-5',
  max_tokens: 150,                  // hard cap = your latency and cost ceiling
  system: [
    { type: 'text', text: PRODUCT_TAXONOMY,
      cache_control: { type: 'ephemeral' } }   // everything above this is cached
  ],
  messages: [{ role: 'user', content: ticketBody }]
})

Tool calling

OpenAI nests the schema under function.parameters; Anthropic uses a flat inputschema. Both accept JSON Schema, but the strictness differs. OpenAI’s structured outputs mode guarantees schema conformance for a constrained subset of JSON Schema. Anthropic gets you there through tool use with high reliability but a different failure mode: the model deciding not to call the tool at all. Handle stopreason properly and you’ll survive either.

Rate limits

This is where teams get hurt. OpenAI tiers you on cumulative spend and gives you requests-per-minute plus tokens-per-minute. Anthropic separates input tokens per minute from output tokens per minute, which is genuinely useful because those are different bottlenecks. A batch job that reads 200 documents will hit ITPM long before RPM. Model that before launch, not during your first traffic spike.

My honest read in 2026: for most application work the capability gap between the mid tiers of the major providers is small enough that it should not drive your decision. Pick on latency profile, rate limit headroom, and whichever one your team’s prompts already work well on.

Where the seconds actually go

Total latency splits cleanly into time to first token (TTFT) and generation time. TTFT covers queueing, prefill of your input, and routing. Generation is output tokens multiplied by inter-token latency. On a typical mid-tier model streaming at 50 to 90 tokens per second, 400 output tokens is four to eight seconds of generation no matter how fast the provider’s infrastructure is.

Consequences worth internalising:

• Cutting your prompt from 8,000 to 4,000 tokens barely moves TTFT once caching is in play. Prefill is parallel and cache hits skip most of it. Teams spend days shrinking prompts for latency and get 100ms back.
• Cutting output from 400 tokens to 80 is a 4x latency win. Ask for structured fields instead of prose. Ask for three bullets, not “a summary”.
• Reasoning models bill their thinking as output tokens and those tokens are generated sequentially too. A model that thinks for 1,200 tokens before answering adds fifteen-ish seconds. Sometimes worth it. Rarely worth it for classification.
• Stream everything user-facing. TTFT of 600ms with streaming feels faster than a 2.5 second non-streamed response, and it’s the same total work.

Measure p95 and p99, not the mean. Provider latency is noticeably spikier at peak hours, and the difference between a 2 second mean and an 11 second p99 is the difference between a feature that works and a support queue. Log ttftms, totalms, output_tokens and the model ID on every call from day one. You cannot reconstruct this later.

Lock-in lives in your prompts, not your SDK

Every “avoid lock-in” article tells you to wrap the API client. Fine, do that, it takes an afternoon. Here’s what nobody says: swapping the HTTP layer is the trivial part. What actually keeps you on a provider is:

1. Prompts tuned to one model family. Two years of accumulated “always respond in this exact format” phrasing that a different model interprets 8 percent differently. That 8 percent is your regression.
2. Fine-tuned weights. Non-portable, full stop. If you’ve fine-tuned, you’ve chosen. That can be the right call, and we’ve written about the cases where fine-tuning still wins, but go in knowing the exit cost.
3. Provider-specific features. Assistants-style hosted state, built-in file search, computer use, extended thinking budgets. Each one is convenience now and a rewrite later.
4. Embeddings. Change embedding models and you re-index your entire vector store. On a corpus of any size that’s a scheduled migration, not a config change. This is the one that surprises people building internal knowledge assistants.

The defence against all four is the same, and it isn’t an abstraction layer. It’s an eval set. Two hundred real inputs with known-good outputs and a scoring function you trust. With that, swapping providers is an afternoon of running evals and a judgement call. Without it, swapping providers is a leap of faith that no sensible engineering manager will approve. That’s what real lock-in feels like from the inside.

Should you use LiteLLM, OpenRouter or the Vercel AI SDK?

Use a thin adapter you wrote. That’s my default position and I’ll defend it.

Router services and universal SDKs are genuinely useful in two cases: you’re prototyping and want to try nine models this week, or you need a single billing relationship across many providers for procurement reasons. Outside those two cases, they add a hop of latency, a second point of failure, an obfuscation layer over the exact usage fields you need for cost attribution, and a lag of days or weeks before new provider features are exposed. Prompt caching support in particular tends to arrive late in abstraction layers, and caching is your biggest cost lever.

A useful adapter is about 80 lines per provider and normalises exactly four things: the text out, the usage numbers, the stop reason, and the error class. Everything else you pass through raw.

// One shape in, one shape out. Provider quirks stay inside the adapter.
export async function callModel({ task, system, messages, maxTokens }) {
  const { provider, model } = ROUTES[task];   // routing table, not hardcoded
  const t0 = performance.now();
  const res = await PROVIDERS[provider]({ model, system, messages, maxTokens });
  logUsage({
    task, provider, model,
    ttftms: res.ttftMs, totalms: performance.now() - t0,
    intokens: res.usage.in, cached: res.usage.cacheRead, outtokens: res.usage.out,
    cost_usd: estimate(model, res.usage)      // attribute cost per task, per tenant
  });
  return res;
}

That logUsage call is the most valuable line in the file. Cost per task per tenant is how you find out that 4 percent of your users generate 60 percent of your spend.

Route by task, and cap everything

The all-frontier default is the most expensive mistake in production LLM work. Most pipelines contain a lot of work that a small model does at parity: intent classification, entity extraction, language detection, routing, reranking, boolean judgements. Push those down a tier and measure. If your evals hold, you’ve just cut that stage’s cost by 90 percent and its latency by half.

A three-tier routing table plus escalation covers most applications:

• Tier 1 (small): classification, extraction, short rewrites, guardrail checks. Sub-second, negligible cost.
• Tier 2 (mid): the default for generation, summarisation, chat. Where 80 percent of your traffic should land.
• Tier 3 (frontier or reasoning): multi-step agentic work, hard code generation, anything where a wrong answer costs real money. Escalate into it on a confidence signal or an explicit user action, never by default.

Then cap. Hard max_tokens on every call. A per-tenant daily token budget enforced in your own code, because provider spend limits are account-wide and blunt. A circuit breaker that falls back to the next provider on 429s and 5xx responses. We treat that fallback as a correctness requirement, the same way we treat backups on a client site: you don’t get to skip it because it has never failed yet.