Observability for LLM Features: Tracing, Cost and Regression

A client called us in March about an OpenAI bill that had gone from roughly $400 a month to $6,200. They had four AI features in production: a support reply drafter, a product description generator, a semantic search layer, and an internal “summarise this thread” button. Nobody could tell us which one was responsible. The provider dashboard showed total tokens by API key, and they had exactly one API key.

It took two days to find it. A retry loop in the summariser was firing three times per request against a 200k-token context because someone had changed the input from “last 20 messages” to “whole thread”. The feature still worked. That’s the part people miss: LLM failures are usually silent, expensive and correct-looking.

LLM observability is the discipline of answering three questions quickly: what happened inside this request, what did it cost and who pays, and did last week’s prompt change make the output worse. Standard APM answers none of them.

LLM observability has three jobs, not one

Teams conflate them and then buy one tool that does a third of each. Separate them deliberately.

Tracing is debugging. When a support agent says “the drafter produced nonsense for ticket 48213”, you need the exact prompt, the retrieved chunks, the model version, the temperature, the raw response and the parse step, within about 30 seconds. This is high-detail, low-retention data. Keep it 14 to 30 days.

Cost accounting is finance. Low-detail, high-retention, aggregated per tenant, per feature, per day. You need this for 24 months because someone will eventually ask what the margin on the AI tier was last Q3. Your traces will be long gone.

Regression is quality control. It runs in CI, not production, against a fixed dataset, and its output is a number you can gate a deploy on. Production traces feed it (that’s where good test cases come from), but it is not the same system.

Build them as three stores with three retention policies. Trying to make one Postgres table serve all three is the mistake we see most often. It dies at around 2 million rows when someone runs a cost report that scans every prompt body.

How to trace LLM calls without drowning in spans

The useful unit is the user action. “Draft a reply” is one trace. Inside it you might have an embedding call, a vector search, a prompt assembly step, a completion, a JSON parse that failed, and a retry completion. Six spans. Instrument only the HTTP call to the provider and you’ll see two completions with no explanation for the gap between them.

Here’s the wrapper we use on Node services. Deliberately thin: no framework, no abstraction over the provider SDK.

import { trace, SpanStatusCode } from '@opentelemetry/api';
import OpenAI from 'openai';

const tracer = trace.getTracer('llm-features');
// Turn OFF the SDK's silent retries so failures show up as spans you can count.
const client = new OpenAI({ maxRetries: 0 });

export async function tracedChat({ feature, tenantId, messages, model }) {
  // Span name follows the GenAI convention: "{operation} {model}"
  return tracer.startActiveSpan(chat ${model}, async (span) => {
    span.setAttributes({
      'gen_ai.operation.name': 'chat',
      'gen_ai.system': 'openai',
      'gen_ai.request.model': model,
      'gen_ai.request.temperature': 0.2,
      'app.feature': feature,       // your own namespace, not gen_ai.*
      'app.tenant_id': tenantId,
      'app.prompt_version': 'reply-drafter@2026-02-11',
    });

    const t0 = performance.now();
    let firstTokenAt = null;

    try {
      const stream = await client.chat.completions.create({
        model, messages, temperature: 0.2, stream: true,
        // Without this, streamed responses return NO usage object at all.
        streamoptions: { includeusage: true },
      });

      let text = '', usage = null;
      for await (const chunk of stream) {
        if (chunk.usage) usage = chunk.usage;
        const delta = chunk.choices?.[0]?.delta?.content;
        if (delta) {
          firstTokenAt ??= performance.now();
          text += delta;
        }
      }

      span.setAttributes({
        'gen_ai.response.model': model,
        'genai.usage.inputtokens': usage?.prompt_tokens ?? 0,
        'genai.usage.outputtokens': usage?.completion_tokens ?? 0,
        'genai.usage.cachedinput_tokens':
          usage?.prompttokensdetails?.cached_tokens ?? 0,
        'app.ttft_ms': firstTokenAt ? Math.round(firstTokenAt - t0) : -1,
        'app.total_ms': Math.round(performance.now() - t0),
      });
      return { text, usage };
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.setAttribute('app.error_type', err.constructor.name);
      throw err;
    } finally {
      span.end();
    }
  });
}

Two things in there matter more than they look. Setting maxRetries: 0 is a position worth defending: the OpenAI SDK retries twice by default, those retries are invisible to your traces while being fully visible on your invoice. Own your retry policy at the application layer where you can span it. And streamoptions.includeusage is the single most common cause of “why are half our requests showing zero tokens” that we get asked about. Streaming responses omit usage unless you ask.

What to put on the span, and what not to

Put on it: model, operation, token counts (input, output, cached, reasoning if the model reports them), finish reason, temperature, prompt version identifier, feature name, tenant ID, TTFT, total duration, retry count, and tool names invoked. For retrieval-augmented features, add the retrieved document IDs and their scores as a child span attribute. When a RAG answer is wrong, it’s wrong because of retrieval the majority of the time, and without the chunk IDs you’ll spend an hour guessing.

Do not put full prompt and completion bodies on spans you send to a metrics backend. Put them in a blob store or a dedicated trace tool keyed by trace ID. And never use tenant ID or user ID as a metric label. We watched a team take down their own Mimir cluster with a llmtokenstotal{user_id=...} counter across 90,000 users.

Cost attribution: your own ledger, not the provider dashboard

Provider dashboards aggregate by key and project. Your business needs cost by feature, by tenant, by plan tier, and per request so you can find the 0.1% of requests burning 40% of the budget. That means a ledger you write yourself.

-- Append-only. One row per model call. Never updated.
CREATE TABLE llm_usage (
  id            bigserial PRIMARY KEY,
  occurred_at   timestamptz NOT NULL DEFAULT now(),
  trace_id      text NOT NULL,
  tenant_id     uuid NOT NULL,
  feature       text NOT NULL,
  model         text NOT NULL,
  prompt_version text NOT NULL,
  input_tokens  integer NOT NULL,
  cached_tokens integer NOT NULL DEFAULT 0,
  output_tokens integer NOT NULL,
  -- micro-dollars as bigint: no float arithmetic on money, ever
  cost_micros   bigint NOT NULL,
  price_version text NOT NULL,
  status        text NOT NULL            -- ok | error | truncated
);

CREATE INDEX ON llmusage (tenantid, occurred_at DESC);
CREATE INDEX ON llmusage (feature, occurredat DESC);

Prices live in a separate versioned table, and every row records which price version computed it. When a provider changes pricing (they do, usually downward, sometimes mid-month) your historical numbers stay reproducible. Store money as integer micro-dollars. A cent is too coarse when a single call can cost 0.0004 dollars.

The query that earns its keep is cost per successful user action, not cost per API call:

SELECT feature,
       count(DISTINCT trace_id)                        AS actions,
       sum(cost_micros) / 1e6                          AS usd,
       (sum(costmicros) / count(DISTINCT traceid)) / 1e6 AS usdperaction,
       sum(cachedtokens)::float / nullif(sum(inputtokens), 0) AS cachehitratio
FROM llm_usage
WHERE occurred_at >= now() - interval '7 days'
GROUP BY feature
ORDER BY usd DESC;

That cachehitratio column has paid for itself repeatedly. Prompt caching is the cheapest optimisation available: at the time of writing, Anthropic bills cache reads at roughly 10% of the base input price (with a write surcharge of about 25%) and OpenAI discounts cached input substantially too. But caching only fires if your static content comes first and is byte-identical. One timestamp injected at the top of a system prompt drops your hit ratio to zero, your cost triples, and nothing breaks. You’d never notice without this column. If the token-and-cost mechanics aren’t second nature yet, our piece on context windows, tokens and cost covers the mental model.

Latency: two numbers, and never the mean

Report time to first token and total completion separately, at p50, p95 and p99. The mean of an LLM latency distribution is close to meaningless. The distribution is bimodal: cached or short-output requests cluster low, long-generation requests sit in a second hump, and a provider degradation event adds a third.

Track output tokens per second as a derived metric. It’s the cleanest early warning for provider-side degradation, because it moves before error rates do. We alert when p50 throughput for a given model drops more than 40% below its own 7-day baseline. That has caught three separate provider slowdowns before users complained.

Also instrument the non-model time. In one audit, a “slow AI search” complaint turned out to be 2.1 seconds of model time and 4.3 seconds of a synchronous embedding call plus an unindexed metadata filter on the vector store. The model was the innocent party.

Catching regression before your users do

Prompts are code without types, tests or a compiler. Every edit is a potential regression, and the failure mode is subtle: slightly more hedging, an occasional dropped field in the JSON, a tone shift that marketing notices three weeks later. You need an evaluation suite in CI, gated like any other test.

Build the golden set from production traces

Do not invent test cases. Pull them from your traces: 60 to 200 real inputs, deliberately including the ugly ones (empty fields, 14-language mixed input, the customer who pastes an entire PDF). Add every case that caused a support escalation. Label expected behaviour, not expected exact strings. For structured output, assert on the schema and on specific field values. That part is deterministic and cheap.

// evals/run.mjs  --  exits non-zero when quality drops. Wire to CI.
import cases from './golden-set.json' with { type: 'json' };
import { tracedChat } from '../src/llm.js';
import { judge } from './judge.js';   // pinned judge model, temperature 0

const BASELINE = 0.87;        // committed score from the last green run
const TOLERANCE = 0.02;       // allow judge noise, block real regressions

const results = [];
for (const c of cases) {
  const { text } = await tracedChat({ ...c.input, feature: 'eval' });
  const hard = c.assertions.every((a) => new RegExp(a).test(text));  // deterministic checks
  const soft = hard ? await judge(c.input, text, c.rubric) : 0;      // 0..1
  results.push({ id: c.id, hard, soft });
}

const score = results.reduce((s, r) => s + r.soft, 0) / results.length;
const failures = results.filter((r) => !r.hard);

console.table(failures.slice(0, 10));
console.log(score ${score.toFixed(3)} vs baseline ${BASELINE});

if (failures.length || score < BASELINE - TOLERANCE) process.exit(1);

Three rules make this trustworthy. Pin the judge model to an explicit dated version and set temperature to 0, or your baseline drifts when the provider silently updates the alias. Keep deterministic assertions doing most of the work, because LLM-as-judge is noisy at the single-case level even if it’s decent in aggregate. Commit the baseline score to the repo so a regression shows up as a diff somebody has to justify in review.

The honest caveat: a 100-case suite cannot reliably detect a quality change smaller than about 5 percentage points. Judge noise swamps it. If you need finer resolution, you need either more cases or human review on a sampled subset, and human review costs real money. Below that threshold, ship and watch production signals instead: thumbs-down rate, edit distance between the draft and what the user actually sent, regeneration rate. Those three are better quality metrics than anything your eval suite produces. They just arrive too late to block a deploy.

What to actually install in 2026

Start with OpenTelemetry. The GenAI semantic conventions exist, Traceloop’s OpenLLMetry and the vendor SDKs emit them, and every serious tool ingests OTLP. Instrument once, change backends later. That’s worth more than any single tool’s feature list.

On top of that, pick by what you’re missing:

• Langfuse if you want self-hosted traces plus prompt versioning plus datasets in one place. The OSS version is genuinely usable, not a crippled demo. This is our default for client work where data residency matters.
• Arize Phoenix if your pain is RAG retrieval quality and you want embedding and cluster visualisation. It runs locally in a notebook, which makes it a good debugging tool even if it’s not your production store.
• Braintrust or LangSmith if the bottleneck is evaluation workflow and getting non-engineers to label data. You’re paying for the UI and the dataset management, and that’s a legitimate thing to pay for.
• Helicone or a gateway if you need cost controls per key today and have no appetite for code changes. A proxy gets you logging and rate limits in an afternoon. It also puts another hop in your latency path and another place to be down, so treat it as a stopgap rather than architecture.
• Grafana plus Postgres for cost dashboards. Your ledger is relational data. Don’t pay a per-event SaaS price to aggregate it.

Two things worth skipping. Anything that requires routing all traffic through a proprietary SDK wrapping the provider client: you lose access to new model parameters on launch day and you’ll be writing an escape hatch within six months. And don’t buy a tool before you have the ledger. The dashboard is the easy part. The schema decisions are what you live with.