LLM Cost & Latency Observability with OpenTelemetry

LLM cost and latency observability starts with recording usage and timing at the model-call boundary. This guide covers the implementation: preserve provider-reported token counts, derive a versioned cost estimate, measure streaming delays, and produce aggregate metrics that do not depend on which traces you retain.

For the attribute vocabulary, use SentryML’s OpenTelemetry GenAI semantic conventions. For spend allocation, see token cost observability; for infrastructure economics, see self-hosting an LLM vs API cost. Here the goal is to produce the records those decisions depend on.

Compute cost yourself; don’t wait for the invoice

Derive a request-level estimate from token usage and reconcile it with provider billing later. Preserve the raw usage separately from the estimate so either can be inspected. The Trace Span Designer can help plan fields; validate them against the convention version used by your exporter.

• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — the raw counts from the provider response.
• gen_ai.request.model and gen_ai.response.model — requested and returned model identifiers, when available.
• A custom app.estimated_cost_usd field derived from a versioned price table. For a simple text-only tariff quoted per million tokens: (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000.

This formula is an illustrative estimate for two usage categories. If your tariff distinguishes cached input, other modalities, or additional charges, preserve those categories and apply the corresponding terms. Record the rate unit, currency, and price-table version used. Keep the original estimate when prices change; any repricing should be labeled as a separate scenario. Missing usage or an unknown tariff should remain unknown, not silently become zero cost.

The latency breakdown that actually helps

“The request was slow” is useless; “the second LLM call’s time-to-first-token spiked” is actionable. Capture latency in parts:

• Time to first token (TTFT) for streaming responses — the number users actually feel. Track it separately from total duration.
• Total generation time and, where you can derive it, inter-token latency — distinguishes a slow start from a slow stream.
• Queue/scheduling time vs provider time — if you have a gateway or rate-limiter in front, time spent waiting in your queue is a different problem than the provider being slow.
• For multi-step requests, per-span latency across the waterfall (retrieval, first LLM pass, tool call, synthesis) so you can see which step regressed. (For the full span structure, see our piece on what belongs in an LLM trace.)

Define each timing boundary explicitly. First response chunk and first visible token need not be the same event; a stream may begin with metadata. Keep queue time and client-observed call duration separate so the trace can show where a delay occurred.

Implement a versioned exporter contract

The GenAI attribute registry provides field and migration references, including the replacement of gen_ai.system by gen_ai.provider.name. The current client-span specification remains in Development. Pin instrumentation and convention versions, then verify the fields reaching your backend. OpenTelemetry transport support alone does not establish which attributes a backend indexes or uses in its views.

# Illustrative exporter fragment; variables are supplied by the application.
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", provider_name)
span.set_attribute("gen_ai.request.model", requested_model)
if input_tokens is not None:
    span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
if output_tokens is not None:
    span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
if estimated_cost_usd is not None:
    span.set_attribute("app.estimated_cost_usd", estimated_cost_usd)
    span.set_attribute("app.price_table_version", price_table_version)
if time_to_first_token_ms is not None:
    span.set_attribute("app.ttft_ms", time_to_first_token_ms)

Slice cost and latency by dimensions that mean something

Aggregate numbers hide the story. The dimensions worth slicing by — while respecting cardinality limits — are:

• Per feature/route — which product surface is expensive or slow. Use a low-cardinality feature_id, never a raw URL.
• Per model — cost and latency profile differs sharply across models; routing decisions live here.
• Per cache outcome — prompt-cache hit vs miss is often the dominant cost lever; a falling hit rate is a cost regression in disguise.
• Per finish reason — a spike in length/truncation finishes inflates output tokens and signals a prompt problem.

Keep metric dimensions bounded. Prefer route identifiers and approved cohort categories to raw user IDs or prompts. Hashing a unique identifier does not reduce its cardinality; trace indexing and retention need their own limits.

Sample without losing the expensive tail

If full trace retention exceeds your budget, use tail-based sampling to select traces using observed duration, errors, or cost. A candidate retention policy might request:

• 100% of errors
• 100% of slow requests (above your p95 latency)
• 100% of high-cost requests (above an explicit app.estimated_cost_usd threshold)
• A baseline sample (e.g., 10%) of normal successful requests
• 100% for the first 24h of any newly deployed feature, then decay

These are example retention targets. The Collector tail-sampling processor supports duration, status, numeric-attribute, and probabilistic policies. Configure thresholds and capacity explicitly; the sampler cannot recover traces dropped before they reach it. Emit aggregate usage and timing metrics before trace sampling, and monitor exporter loss. A retained subset biased toward slow requests is not a representative latency distribution or a total-spend ledger.

Reconcile call usage with request totals

Keep one accounting record per logical model call and identify its parent request. Retries may incur additional usage; capture it where the provider reports it without counting the same usage again on both parent and child spans. Add a coverage metric for calls whose final usage is missing, including interrupted streams.

Display usage coverage beside the cost panel in the ML model monitoring dashboard. Use a complete metrics stream for totals and retained traces for investigation. Attach online evaluation scores through request identifiers so a cost reduction can be examined alongside output quality.

Alert on the two numbers that kill features

Close the loop with alerts tied to the business reality:

• Cost per request, per feature, with a hard ceiling and a trend alert. A doubling almost always traces to one cause: a prompt that grew, a cache hit-rate drop, output-length creep, or a model swap. Your slices answer “which” in seconds.
• Tail latency (p95/p99 TTFT), per feature, against an SLO. Page on sustained breach, not a single spike.

These two alerts catch the regressions that otherwise surface as an end-of-month invoice surprise or a quiet drop in feature usage — both of which are far more expensive to discover late.

The payoff

Instrument cost and latency to the OTel GenAI conventions and the questions that used to mean a war room become queries: “why did this feature’s cost double last Tuesday?” → compare token-per-call and cache-hit-rate across the two days. “Why is the assistant slow for some users?” → TTFT-by-step on the slow-tail traces. The instrumentation is a few attributes per span and a price table you maintain. The return is that the two numbers most likely to get your LLM feature killed are the two you can see coming.

See also

• End-to-end tracing for LLM applications: what belongs in a span
• The open-source ML observability stack: Evidently to Phoenix
• Alerting for ML model drift: a practical setup
• ML observability: layers, signals, and architecture
• ML model monitoring dashboard: what to put on it