A client bot we inherited last year was billing about $4,200 a month on a product that had roughly 900 active users. Nobody could explain it. The prompt was 600 tokens, the answers were short, and the traffic numbers didn’t justify anything close to that. The problem turned out to be one line of code: every turn, the app sent the entire conversation history back to the model, including a 14,000 token product catalogue that had been pasted into the system prompt “so it always has context”.

That bug is the most common one in production LLM apps, and it isn’t really a bug. It’s a missing mental model. Most developers think of a context window as a storage limit, like a file size cap: stay under it and you’re fine. It’s closer to a metered utility you pay for on every single request, at a rate that compounds with conversation length.

Here’s the model that holds up once you’re paying real invoices.

A token is not a word, and the gap costs money

Tokenizers split text into subword units using byte pair encoding. GPT-4o and the o200k_base tokenizer land around 4 characters per token for ordinary English prose, which is where the “750 words per 1,000 tokens” rule of thumb comes from. That rule falls apart fast.

Minified JSON with long key names tokenizes badly. UUIDs tokenize horribly, often one token per 2 to 3 characters. Non-Latin scripts can be 2 to 4 times more expensive per character than English. A base64 blob is close to worst case. We once shaved 31% off the input size of a classification pipeline purely by renaming JSON keys from customershippingaddresslineone style to short codes and dropping the indentation before sending.

Stop guessing. Count:

import tiktoken

o200k_base is the encoding used by GPT-4o and GPT-4.1 family models.
enc = tiktoken.getencoding("o200kbase")

prose = "The quick brown fox jumps over the lazy dog."
blob  = '{"customer_identifier":"8f14e45f-ceea-467a-9e41-ff2d4b07a1d9"}'

print(len(enc.encode(prose)))  # 10
print(len(enc.encode(blob)))   # 31 tokens for 62 characters

Every provider tokenizes differently. Anthropic exposes a token counting endpoint; Google exposes countTokens. If you’re multi-provider, count per provider, or your budget guards will be wrong by 10 to 20% in the direction you don’t want.

The context window is a budget, not a feature

As of early 2026 the headline numbers are large: GPT-4.1 ships a 1M token context window, Gemini 2.5 Pro does 1M, Claude’s Sonnet tier is 200K with a 1M tier available. Marketing treats these as capacity. Treat them as credit limits.

Three things share that budget on every request: your system prompt, the accumulated conversation or retrieved documents, and the space reserved for the output. That last one trips people up. Fill 199,000 tokens of a 200K window and you have not left room for a 4,000 token answer. You’ll get a truncation error or a cut-off response depending on the SDK.

The rule we use on client builds: decide the output reservation first, subtract it, then allocate what’s left between the fixed prefix and the variable context. Write it down as actual constants in code, not as an assumption in someone’s head.

const WINDOW = 200_000;
const OUTPUTRESERVE = 4000;      // hard ceiling on max_tokens
const SAFETY = 2_000;              // tokenizer drift between providers
const PREFIX = 6_200;              // system prompt + tool schemas, measured

// Everything else (history, RAG chunks) must fit here.
const CONTEXTBUDGET = WINDOW - OUTPUTRESERVE - SAFETY - PREFIX; // 187,800

Why linear conversations produce quadratic bills

Chat APIs are stateless. Every turn you resend everything. Turn one bills 800 tokens, turn two bills 1,600, turn three 2,400, and so on.

Run the arithmetic on a 40 turn support conversation with 800 new tokens per turn. Total billed input is 800 × (40 × 41 / 2) = 656,000 tokens. The unique content was 32,000 tokens. You paid for 20.5 times what you wrote. At a mid-tier rate of roughly $3 per million input tokens, that’s about $1.97 per conversation against a theoretical floor near $0.10.

Multiply by 900 users having two conversations a week and the $4,200 invoice stops being mysterious.

Three mitigations, in the order we apply them:

1. Cache the stable prefix so the system prompt and tool schemas stop being charged at full rate.
2. Trim or summarise history past a threshold, keeping the first turn (which usually carries the task framing) and the last N turns verbatim.
3. Stop sending documents in the prompt when retrieval against an index would send 2,000 tokens instead of 14,000.

Prompt caching is the lever that actually moves the number

This is the single highest return change available to most teams, and plenty of production apps still don’t use it. Providers cache the key-value attention state for a prompt prefix so that identical prefixes on subsequent requests skip the prefill compute.

On Anthropic’s API, a cache write costs about 1.25 times the normal input rate and a cache read costs about 0.1 times, with a default five minute TTL and a longer option. OpenAI applies automatic caching to prefixes above a minimum length and discounts those cached tokens substantially. The economics are blunt: if your 6,200 token system prompt is sent 50 times in five minutes, caching takes it from 310,000 billed tokens to roughly 7,750 write plus 30,400 read equivalent.

The catch: caching is prefix-based and exact-match. One dynamic value near the top of your prompt destroys it.

const messages = [
  {
    role: "system",
    content: [
      {
        type: "text",
        text: SYSTEMPROMPT + TOOLDOCS,     // byte-identical on every call
        cache_control: { type: "ephemeral" } // marks the cache breakpoint
      }
    ]
  },
  // Anything volatile (timestamps, user name, session id) goes AFTER
  // the breakpoint, never before it.
  { role: "user", content: Today is ${today}. ${userMessage} }
];

We’ve seen teams put Current time: 14:32:07 as line one of the system prompt and wonder why their cache hit rate sits at zero. Log cachereadinputtokens against inputtokens and alert when the ratio drops. It’s a two line dashboard that pays for itself.

More context often means worse answers

The uncomfortable finding from the “lost in the middle” line of research, one that still reproduces on 2026 frontier models, is that retrieval accuracy follows a U-shape across position. Information at the start and end of a long input is recalled reliably. Information buried at 40 to 60% depth is recalled much less reliably, and the effect gets worse as total length grows.

Vendor needle-in-a-haystack charts show near-perfect recall because they test a single distinctive fact. Real workloads need multi-hop reasoning over several facts that look similar to each other. Performance there drops off well before the window is full.

Our working heuristic: if a task is accuracy-critical, don’t exceed about half the advertised window, put the instruction both before and after a long document, and prefer 8 well-chosen retrieved chunks over 80 mediocre ones. Dumping an entire codebase into a 1M window and asking for a bug fix is usually the wrong choice. It’s slower, costs 50 times more, and tends to produce worse output than a focused prompt with the three relevant files. If you want to prove that for your own workload rather than taking our word for it, build a lightweight quality harness and run both configurations against the same 40 cases.

Output tokens, reasoning tokens, and the invisible bill

Input tokens are the cheap half. Output is typically 3 to 5 times the input rate across providers, because generation is sequential and memory-bandwidth-bound while prefill is parallel and compute-bound.

Reasoning models complicate this. The thinking tokens a model generates before its visible answer are billed as output, and they can easily be 5 to 10 times the length of the answer you see. A request returning a 200 token response might have generated 3,000 reasoning tokens. If your cost model only counts visible characters, you’ll be out by an order of magnitude.

Two habits fix most of it. Set a real max_tokens rather than leaving it at the SDK default. Then ask for terse output explicitly: on a classification task, “respond with one of: refund, shipping, technical” beats a free-form answer on cost, latency and parse reliability all at once. This matters doubly inside agent loops, where each tool call adds its result to the context permanently. See our notes on agent architecture and failure modes for how fast that compounds when a tool returns an unbounded API payload.

Instrument it or you’re guessing

Every provider returns a usage object. Log it, every call, with a route label. The schema we use on client projects is deliberately boring:

CREATE TABLE llm_calls (
  id            bigserial PRIMARY KEY,
  created_at    timestamptz NOT NULL DEFAULT now(),
  route         text NOT NULL,          -- 'support.reply', 'summarise.doc'
  model         text NOT NULL,
  input_tokens  integer NOT NULL,
  cached_tokens integer NOT NULL DEFAULT 0,
  output_tokens integer NOT NULL,
  reasoning_tokens integer NOT NULL DEFAULT 0,
  latency_ms    integer NOT NULL,
  cost_usd      numeric(10,6) NOT NULL  -- computed at write time
);

Compute cost at write time rather than in reports, because prices change and you want the historical number to stay honest. Then answer one question weekly: which route has the highest cost per successful outcome? Not per call. Per outcome. A route that costs $0.004 per call but needs three retries is worse than one at $0.009 that works first time.

Set a per-request hard cap too. A token budget guard that refuses to send a request above, say, 60,000 tokens will catch the runaway loop before it catches your card. We’ve had an agent recursively include its own output and burn through four figures in under an hour. Once.

When a huge window genuinely earns its keep

There are real cases. Single-pass analysis of a long document where chunk boundaries would destroy meaning, like a 300 page contract where clause 14 modifies clause 2. Transcript work where speaker continuity matters. Migration tasks where you need the old file and the new schema side by side. Codebase-wide refactors where the dependency graph genuinely is the context.

What those share is that the content is read once and discarded, not accumulated across turns. A large window is excellent for one-shot breadth and dangerous for long-lived conversations. If you’re running self-hosted, the calculus shifts again, since the cost is GPU memory for the KV cache rather than per-token billing, and that has its own practical limits.