End-to-End Tracing for LLM Apps: Span Design Guide

An LLM request may include retrieval, model calls, tools, retries, and output validation. End-to-end tracing for LLM applications connects those operations to one request so an investigation can follow their timing and recorded evidence. If you are still choosing signals, start with how to monitor an LLM in production. The cost and latency implementation guide adds usage accounting to the same request path.

This guide focuses on span boundaries, propagation, and investigation checks. For the vocabulary and its scope, use SentryML’s OpenTelemetry GenAI semantic conventions and OpenInference vs OpenTelemetry. The Trace Span Designer can help plan fields; check its output against the convention version your instrumentation emits.

The waterfall

A typical RAG-with-tool-use request decomposes into spans like:

[parent: handle_request]
  ├─ [retrieve_context]
  │   ├─ [embed_query]
  │   └─ [vector_search]
  ├─ [first_pass_llm]   <- planning step
  ├─ [tool_call: search_docs]
  │   └─ [http_request]
  ├─ [second_pass_llm]  <- synthesis step
  └─ [validate_output]

Traces are one of the signal families in the ml observability architecture. OpenTelemetry’s trace model connects spans through trace and parent identifiers. Each child records a timed operation. The parent measures elapsed request time; overlapping child operations must not be summed as if they ran sequentially.

Verify the model-call span at the request boundary

The GenAI client-span specification is marked Development. Pin the convention and instrumentation versions together. Inspect an exported span before building queries around its attributes.

Check the operation and provider fields, requested and returned model identifiers, and input/output token counts when the provider supplies them. Current field names include gen_ai.provider.name, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens; finish reasons use the array field gen_ai.response.finish_reasons. Keep custom cost estimates in an application namespace rather than inventing a standard gen_ai.* field.

The logical model-call span should cover the complete call, including streaming completion and automatic retries. Lower-level transport spans can expose individual attempts. Keep a request identifier in application records so usage, errors, and evaluation results can be joined without counting the same logical call twice.

Leave message content capture off unless an approved investigation or evaluation process requires it. Sampling alone does not remove sensitive content. Prefer version references and restricted records over copying prompts or retrieved documents into general telemetry.

What belongs in a tool-call span

When the model calls a tool, capture:

• Tool name
• Argument hash (full args at low sample rate; hash at full rate to detect duplicates)
• Tool latency (separate from network latency if the tool wraps an HTTP call)
• Tool error class (validation failure, downstream service error, rate limit, timeout)
• Whether the tool’s response was used in the final output (sometimes the model ignores tool output)

Record whether the application passed a tool result into a later model call when that is observable in your code. Whether the answer actually used the result is an evaluation question; absence of a quotation is not evidence that the model ignored it. Online evaluation covers attaching quality scores to request records.

What belongs in a retrieval span

For RAG retrievals:

• Query text (sample-able)
• Number of results returned
• Latency split: embedding vs vector-search vs re-rank
• Similarity scores at the top-k boundary (the marginal kth-result score is the signal you tune cutoffs against)
• Cache hit/miss
• Index version (when did we last re-embed)

If your retrieval system has a tiered cache, capture the tier each result came from. This is where most retrieval-latency regressions hide.

What does NOT belong in a span

• Full prompts and completions at default sampling rates (privacy + storage cost)
• Vector-space embeddings as serialized arrays (huge, low-signal)
• Internal prompt-rendering details that don’t affect output (e.g., template-engine intermediate state)
• Reasoning traces from chain-of-thought models if you don’t expose them to users (they’re useful for debugging but enormous; sample at <1%)
• Anything that would let an attacker reconstruct the system prompt from your traces

Choose searchable fields deliberately

Indexing and retention behavior depends on the trace backend. As a design rule, keep routine filters bounded and keep per-request identifiers out of metrics labels. Hashing a unique identifier changes its representation, not the number of distinct values.

Strategies:

• Use approved cohort fields for routine filters instead of raw user identifiers.
• Retain span timing for latency analysis; use bounded categories only where aggregation calls for them.
• Use semantic equivalents: feature_id instead of feature URL; provider instead of full endpoint.
• Prune URL query strings before capture; keep the path and a hash of the params if you need replay.

Review the selected backend’s indexing and retention settings before estimating storage. Retrieval evidence has its own design requirements; see embedding and vector-store observability.

Sampling

If retaining every trace exceeds your budget, define a sampling policy. An illustrative policy could retain:

• 100% of error requests (decide at end-of-trace)
• 10% baseline of successful requests
• 100% of slow requests (anything > p95 latency)
• 100% of high-cost requests (anything in top decile of cost-per-request)
• Per-feature override for newly deployed features (100% for first 24h, then drop)

These percentages are examples to size against your traffic. The Collector tail-sampling processor supports policies for status, duration, numeric attributes, and a probabilistic baseline. All spans for a trace must reach the same Collector instance. Choose an explicit decision wait and capacity; traces dropped earlier or spans arriving too late may be unavailable to the decision. Percentile-based thresholds need to be computed separately and supplied as policy values.

Attribute propagation

Across services you instrument, preserve the active trace context using the OpenTelemetry propagation model. HTTP instrumentation commonly uses W3C traceparent; verify injection and extraction at both ends. A third-party service that does not participate will not expose its internal spans just because your client sends a header.

When trace context is dropped, you get fragmented traces — the parent span shows in your backend, but the child doesn’t link. Debugging across the gap requires manual ID-stitching, which is error-prone.

Verify the exported request path

Before connecting alerts, exercise a successful request, a failed tool call, a retry, and a streaming cancellation in your own controlled environment. Check that each request has a parent span, expected child operations, useful error classification, and a completed duration. For asynchronous work, verify propagation through the task boundary instead of assuming a shared process preserves context.

Open the trace from the ML model monitoring dashboard using the same route and time window. The open-source ML observability stack covers the roles of the instrumentation and backend components.

What the trace lets you do

Once instrumented, use the retained evidence to investigate questions such as:

• “Why was this user’s request slow?” → load trace, look at the waterfall, identify the long span
• “Why did this feature’s cost double?” → compare last week’s traces to this week’s, look at average tokens per call
• “Why did the model call the wrong tool?” → load the trace, look at the span where the tool call decision was made, inspect input
• “Why did the response contain stale data?” → trace shows retrieval span; check cache hit + index version

An incomplete trace may only narrow the cause. Check missing spans, sampling decisions, and retention before treating an absent operation as proof it never ran.

What to instrument first

If you’re starting from zero, instrumentation order:

1. The outermost request boundary (handle_request span)
2. Each LLM call (with the conventions above)
3. Each tool call
4. Each retrieval
5. Errors (caught exceptions become span events)
6. Cache lookups (separate span, even if cheap)
7. Output validation / classification

Validate each boundary before adding the next. The goal is a connected request record whose gaps are known, with a link back to the release and experiment that produced it.

Cross-references

For token attribution and spend reporting, see token cost observability on LLMOps Report. For the experiment record behind a release, W&B vs MLflow vs Comet compares tracking workflows and their connection to production evidence.

The traces are the foundation; everything else builds on them.