Trace Span Designer

Which store answered — separates a Pinecone outage from a Weaviate one.

Recall ceiling; a low top_k is the most common silent cause of 'the answer wasn't in context'.

The actual query embedded — without it you cannot reproduce a bad retrieval.

How many chunks actually came back (≠ top_k when the index is small or filtered).

Lets you replay the exact context and diff it across versions.

Score floor — a collapsing min score is an early drift signal for the index.

Metadata filter applied; a wrong filter looks identical to 'no relevant docs'.

Query/index embedding mismatch is a classic invisible failure.

Reranker identity — quality regressions usually track a model/version change.

Candidates in; with output.count it reveals how aggressive the cut was.

What actually reached the prompt — the real context budget.

Confidence of the kept set; a low top score after rerank means retrieval failed upstream.

Did rerank actually reorder, or is it dead weight/latency?

Lets you audit a relevant doc the reranker wrongly discarded.

Top-vs-cut score gap — a narrow gap means the ranking is barely meaningful.

Which template/version produced this prompt — the #1 thing to correlate quality changes against.

Final assembled size; silent context truncation is a top cause of dropped answers.

Explicit truncation flag so a cut context is loud, not silent.

Injected variables — needed to reproduce the exact prompt deterministically.

How many chunks survived into the prompt vs how many rerank emitted.

Detects an unannounced system-prompt change across deploys.

Conversation depth — long histories crowd out retrieved context.

Provider — first split for latency/error/cost dashboards.

Requested model; silent model swaps explain sudden quality/cost shifts.

Model actually served — providers alias and roll versions under you.

Cost and context-pressure driver.

Cost driver and a proxy for runaway/verbose generations.

'length' here = truncated answer; the single most actionable quality field.

Reproducibility and explaining nondeterministic regressions.

Too-low caps cause 'length' truncation — pair with finish_reason.

Provider request id for support tickets and dedup.

Prompt-cache hit rate — large lever on cost and latency.

Which tool ran — agents fail at the tool boundary far more than in the model.

Args the model produced — most agent bugs are malformed/hallucinated arguments.

ok/error/timeout — separates a model mistake from a downstream failure.

Correlates the request with the result message fed back to the model.

A truncated tool result silently degrades the next model turn.

Retry loops are a top hidden latency and cost sink in agents.

Position in the agent loop — pinpoints where a multi-step plan derailed.

Privilege of the call — flags an agent exceeding its intended scope.

Which guards ran — a disabled guard is otherwise invisible.

Pass/fail outcome; the headline reliability signal for the stage.

pass/block/redact/regenerate — what the user actually saw.

Failure category — drives a violation-type dashboard, not just a count.

Confidence/groundedness score for tuning the threshold instead of guessing.

Regeneration loops are a hidden cost and latency multiplier.

Guards on the response path add user-visible latency.

Audit trail for what was scrubbed before the user saw it.

Names the pipeline so traces are groupable by product flow, not endpoint.

Stitches multi-turn conversations and reproduces a user's whole journey.

The number the user feels — the SLO that matters.

End-to-end success/failure independent of any single stage.

The only direct quality label most prod systems get — attach it to the trace.

Rolled-up request cost for per-feature/per-tenant economics.

Correlates a quality/cost shift to a specific release.

Per-tenant SLOs, cost, and abuse isolation.