ML Observability: Architecture and Signals

A web service tells you it is broken. It throws a 500, the error rate spikes, and someone gets paged. A machine learning model does not do that. It returns HTTP 200, at normal latency, with a confidently wrong answer, and it will keep doing that for weeks. That gap is the entire reason ML observability exists as a discipline separate from application monitoring.

The word gets used loosely enough to be nearly meaningless, so here is a definition worth holding onto: ML observability is the property that you can explain a production model’s behaviour from the telemetry it already emits, without shipping new code to investigate. Monitoring tells you a number crossed a line. Observability is whether, once that alarm fires, the instrumentation you already have can get you to a cause. Most teams have the first and assume it implies the second; the two disciplines, their costs, and which failures each one catches are set side by side in ML observability vs monitoring.

Why ML breaks the usual assumptions

Three properties of ML systems defeat instrumentation designed for services.

Failure is silent. There is no exception to catch. The failure mode is a distribution of slightly worse predictions, and no individual prediction is identifiably wrong. You cannot alert on something you cannot see in a single request.

Ground truth arrives late, or never. A recommendation click resolves in seconds. A credit decision resolves in months. An LLM answer’s correctness may never resolve at all. Every accuracy-based signal is bounded by that label lag, which means an observability design that depends only on realized performance has a detection floor equal to your slowest label.

The failure is often upstream. A schema change, a unit switch from cents to dollars, a nulled-out feature after an ETL migration — the model is fine and its inputs are not. Instrumenting only the model misses most of what actually breaks it.

Those three facts drive the whole architecture. You instrument the inputs because the outputs are late. You instrument the outputs because the inputs do not tell you whether anything actually got worse. And you instrument the path between them because that is where the debugging happens.

The four layers to instrument

Think of ML observability as four layers, each answering one question, each with its own storage and cadence.

Layer 1 — Data and inputs. “Is the world still the world the model was trained on?” Per-feature distribution summaries computed on a rolling window and compared against a versioned reference: quantiles, null rates, cardinality for categoricals, and a divergence statistic such as PSI, KS, Wasserstein, or Jensen-Shannon depending on the feature type. The Evidently drift-metric documentation is a good map of which test suits which data shape. Add schema and freshness checks here too: a feature that stopped updating is a more common outage than a subtle distribution shift, and it is trivially detectable.

Layer 2 — Model outputs and performance. “Is it still right?” Realized accuracy, AUC, F1, or RMSE on whatever slice has received labels, plus the label-free substitutes you need while you wait: prediction-distribution drift, confidence-score distribution, output entropy, and estimated performance from a method such as NannyML’s confidence-based estimation. Calibration belongs here as well; a model whose ranking is intact but whose probabilities have drifted will pass an AUC check and still break every downstream threshold that was tuned on those probabilities. This layer is covered in depth in model drift detection.

Layer 3 — Serving and infrastructure. “Is it up, fast, and affordable?” p50/p95/p99 latency, throughput, error rate, queue depth, GPU utilization, and for generative workloads time-to-first-token, tokens per second, and cost per request. This is the layer that maps cleanly onto ordinary application monitoring, which is exactly why teams over-invest in it: it is the easy layer, and it is almost never where the interesting failure lives. The instrumentation pattern for the generative side is in LLM cost and latency observability with OpenTelemetry.

Layer 4 — Semantics and quality. “Is the output any good?” For classical ML this layer is mostly empty. For LLM and RAG systems it is the whole ballgame: retrieval relevance, groundedness, refusal rate, toxicity, schema-validity of structured output, and scored evaluations run against live traffic. Embedding and vector-store health sits here too, and it is the layer teams most often ship without any instrumentation at all — see embedding and vector-store observability.

A useful diagnostic: name the last three production incidents your ML systems had, and label which layer would have caught each. Most teams discover they have thorough coverage of layer 3 and almost nothing on layers 1, 2, and 4.

The three signal families

Cutting the same system the other way, ML telemetry comes in three shapes, and conflating them is a common design error.

Distributions. Aggregates over a window, compared against a reference. Cheap to store, cheap to compare, and the only practical way to watch high-cardinality feature spaces. Distributions are how layers 1 and 2 are actually implemented. The critical detail is that the reference window must be versioned and pinned: comparing this week against last week hides gradual drift completely, because the baseline drifts along with the data.

Traces. Per-request causal records: which retriever ran, what it returned, which model was called with which prompt, what it cost, how long each step took. Traces are the debugging substrate. When a distribution alarm fires, the trace is what turns “PSI on feature 14 is 0.31” into “the upstream service started returning nulls at 14:20.” Emit them against the OpenTelemetry signal model and, for generative systems, the GenAI semantic conventions, so the data stays portable across vendors. Deciding what belongs on each span is its own design problem, worked through in end-to-end tracing for LLM applications and buildable interactively with the Trace Span Designer.

Evaluations. Scores attached to individual outputs, produced by a rule, a model-graded judge, or a human. Evaluations are the only signal that directly measures quality when labels never arrive, which makes them indispensable for generative systems and the reason the offline eval suite has to follow the model into production. That pipeline is covered in closing the eval-prod gap.

Distributions detect. Traces explain. Evaluations judge. A stack missing any one of the three has a predictable blind spot: no distributions means late detection, no traces means alarms nobody can action, no evaluations means quality is measured by proxy forever.

ML observability reference architecture

The wiring is more standard than the vendor landscape suggests. Five stages:

1. Emit. The serving path writes a prediction record — inputs or their hashes, output, model version, timestamp, request id — and a trace span. Do this synchronously to a buffer, asynchronously to storage. Observability must never sit in the latency path of the prediction.
2. Land. Prediction records go to columnar storage partitioned by day and model version. Traces go to a trace backend. Operational metrics go to a time-series database. Three stores, because the query patterns are genuinely different, and trying to force traces into Prometheus or metrics into a data warehouse is a well-trodden dead end.
3. Join. Labels arrive later on their own schedule and get joined to prediction records by request id. This job is the one that quietly breaks most often, and it deserves its own freshness monitor: an accuracy dashboard fed by a stalled join looks perfectly healthy.
4. Compute. A scheduled job reads the window, computes drift statistics against the pinned reference, computes realized and estimated performance, runs online evaluations on a sample, and writes the results as metrics. This is the layer where Evidently, NannyML, or a hand-rolled job all slot in interchangeably.
5. Act. Thresholds on those computed metrics feed alerts and a dashboard. Every alert carries a link to the trace query that scopes the affected slice, which is what makes the difference between an alert and a page someone dreads.

Model version belongs in every record at every stage. Without it, a metric change during a rollout is unattributable, and rollout is exactly when things change.

Connect experiments, dashboards, and request evidence

Use the architecture to preserve a path from a production symptom to the release that introduced it. An experiment tracker records the run’s parameters, metrics, and artifacts; MLflow’s tracking documentation describes that record. Keep the deployed artifact’s run identifier in your release metadata, then expose that lookup from the dashboard. A training score and a live quality score describe different datasets, so label both before comparing them.

The following is a suggested navigation design. Grafana dashboard links can carry the selected time range and template variables into another dashboard.

Starting question	First view	Next evidence
What changed with this release?	Model-version comparison	Experiment run and artifact version
Which cohort lost quality?	Performance panel with label coverage	Prediction records for that cohort and window
Which step became slower?	Route latency panel	Request trace with retrieval, model, and tool spans

Build the first view with the ML model monitoring dashboard specification. Use W&B vs MLflow vs Comet for the experiment record behind a release. For generative systems, how to monitor LLMs in production connects the serving and quality signals; online evaluation explains how to attach production scores.

What to instrument first

Ordered by value per hour of work, for a team starting from nothing:

1. Log every prediction with inputs, output, model version, and request id. Nothing else is possible without this, and it is the one thing that is painful to backfill.
2. Null rate and freshness per input feature. Catches the most common real outage, needs no statistics, and produces almost no false positives.
3. Prediction-distribution monitoring. One time series per model, alerting on shift. Label-free, immediate, and it catches a startling share of real problems.
4. Realized performance on whatever labels you have, segmented by the two or three business dimensions that matter.
5. Feature-level drift on the top ten features by importance. Not all 400. Ranking the alerts by importance is what keeps the channel readable.
6. Traces, once the alerts start firing and nobody can explain them.

That order is deliberate: it front-loads the cheap signals that fail loudly and defers the expensive ones until there is a question they answer. It is also the order that makes an investigation converge: each item corresponds to a hypothesis in the triage path in debugging model accuracy drops in production, which is the payoff for doing this work before an incident rather than during one.

The failure modes that recur

Aggregate-only metrics. A model holding its overall number while one cohort collapses still reports as healthy, because the aggregate is a volume-weighted average and the failing cohort is usually the small one. Slice every performance panel by cohort, or the average will hide the incident that matters.

A rolling reference window. Comparing against last week means gradual drift is structurally invisible, because the baseline moves with the data. Pin the reference to the training distribution and version it alongside the model.

Alerting on every feature. Four hundred drift alarms is not observability, it is a channel people mute. Alert on importance-weighted drift and on output-side signals; leave the rest as dashboard context. The threshold and routing design is worked through in alerting for ML model drift.

No owner. A dashboard nobody is accountable for reading decays into decoration within a quarter. Every panel needs a name attached and an action defined for when it goes red.

Vendor-shaped instrumentation. Wiring traces to a proprietary SDK rather than an open standard converts a tool re-evaluation into a re-instrumentation project. Emit OpenTelemetry and let the vendor consume it.

Build or buy

The layers above are the requirement; the tooling is an implementation detail you can defer. An open-source stack assembled from Evidently, Prometheus, Grafana, and Phoenix covers all four layers competently and is described in the open-source ML observability stack. If experiment tracking and registry are part of the same decision, the hosting-and-data-residency framing in Weights & Biases vs MLflow vs Comet is the question to answer before comparing feature lists. Commercial platforms mostly buy you the compute-and-act stages pre-built, which is worth real money when the alternative is a team maintaining scheduled jobs, and worth nothing if you have not yet done step one and logged your predictions.

See also

• ML observability vs monitoring: what actually differs
• Debugging model accuracy drops in production
• Model drift detection: catching performance decay early
• How to monitor an LLM in production
• ML model monitoring dashboard: what to put on it
• The open-source ML observability stack