ML Model Monitoring Dashboard: What to Put on It

Most ML monitoring dashboards are built out of whatever was easy to plot. Request counts, because the web framework already exported them. Average latency, because the metric existed. A model-version pie chart, because it looked good in the demo. None of those panels change a decision, which is why the dashboard gets opened during onboarding and then never again.

A better constraint to design under: every panel must answer a question whose answer would change what someone does today. If a panel goes red and the honest response is “huh, interesting,” it is decoration. Delete it or demote it to a drill-down. Grafana’s own dashboard best-practice guidance makes the same point from the general-purpose side: a dashboard is for answering a specific question for a specific audience, not for displaying everything available.

This is a panel-by-panel spec for an ML model monitoring dashboard that survives contact with a real on-call rotation.

Four rows, in this order

The layout matters more than the panel choices, because the order encodes triage. Someone arriving from an alert should be able to walk down the page and eliminate causes as they go, in the same cheapest-first order used in debugging model accuracy drops in production.

Row 1 — Is it up and fast? Serving health. Cheapest to check, most likely to be the actual cause of a user complaint, and the row that rules itself out in five seconds.

Row 2 — Is it still right? Model quality. Realized performance where labels exist, estimated performance where they do not.

Row 3 — Are the inputs the same? Data and drift. The leading indicator that explains row 2 before row 2 has enough labels to move.

Row 4 — What is it costing? Spend and efficiency. Optional for a classical model on a CPU box, mandatory the moment a token-billed API is in the path.

Put the model version and the reference-window version in the header as template variables, not buried in a panel. Almost every confusing dashboard reading traces back to someone comparing two windows that straddled a deployment.

The panel spec

Each panel below is listed with the metric it shows, where the number comes from, an illustrative threshold, and a possible response. These numerical thresholds are design examples, not vendor defaults or validated production limits. Set your own values using the service objective, a representative baseline, sample size, and the cost of a false alarm. An alert should start an investigation; it should not trigger retraining solely because a number moved.

Row	Panel	Metric	Source	Starting threshold	Action when it trips
1	Latency distribution	p50 / p95 / p99	Prometheus histogram	p99 above SLO for 10 min	Scale, or check batch-size and queueing
1	Error and timeout rate	5xx + client timeouts	Serving logs	Burn-rate on the SLO error budget	Page; treat as a service incident
1	Throughput	Requests per minute	Prometheus counter	Deviation from weekly seasonal band	Check upstream caller, not the model
2	Realized performance	AUC / F1 / RMSE on labeled window	Prediction-label join	2 points below validation baseline	Investigate cohorts, consider retrain
2	Estimated performance	Confidence-based estimate	NannyML-style estimator	Estimate below realized baseline	Check calibration and available labels
2	Performance by cohort	Same metric, sliced	Prediction-label join	Any cohort 10+ points below aggregate	Segment-specific fix, not a global retrain
2	Calibration	Reliability curve, ECE	Prediction-label join	ECE drift beyond baseline	Recalibrate; audit downstream thresholds
3	Prediction drift	PSI on output distribution	Scheduled drift job	PSI above 0.2	Label-free alarm; look at row 3 features
3	Feature drift, top 10	PSI or KS per feature	Scheduled drift job	PSI above 0.2, or KS p below 0.05	Trace the feature to its upstream owner
3	Null and freshness	Null rate, last-updated age	Feature pipeline	Any change from steady state	Page the data pipeline, not the ML team
3	Schema and cardinality	Type changes, new categories	Feature pipeline	Any unexpected value	Block the deploy or quarantine the slice
4	Cost per 1k predictions	Derived from token counts	Trace attributes	20% week-over-week rise	Check prompt length, retrieval size, model tier
4	Token and context usage	Input/output tokens, p95	Trace attributes	Context near model limit	Truncation and retrieval-budget review

Thirteen panels is roughly the ceiling for a screen someone will actually scan. Anything beyond that belongs on a drill-down linked from the panel that motivates it. Note also what a dashboard structurally cannot do: every panel here answers a question written in advance, so the drill-downs behind them have to be queries over per-request records rather than more panels. The reason that distinction decides your architecture is in ML observability vs monitoring.

Panels that look useful and are not

Average latency. An average hides the tail that users experience and that autoscaling reacts to. Prometheus’s histogram guidance is explicit about quantiles over averages, and it is doubly true for inference, where batching produces a bimodal distribution that no mean describes.

Raw feature histograms for every feature. Two hundred sparklines is a wall, not a signal. Rank by feature importance, show the top ten, and put the rest behind a search box.

Aggregate accuracy alone. The single most misleading number on any ML dashboard. A model that is fine in aggregate and broken for one cohort is a broken model, and the aggregate panel is the reason nobody noticed for six weeks.

A rolling week-over-week baseline. If the reference window moves with the data, gradual drift is invisible by construction. Pin the reference to the training distribution, version it, and label the panel with which version it is comparing against. This is the single most common design defect in home-built drift dashboards, and the Evidently drift documentation is worth reading on how test choice interacts with window size and sample count.

Model-version distribution as a pie chart. During a canary it is a bar that should move; outside a canary it is a constant. Neither case is worth permanent screen space. Put version in the header filter instead.

Alerting off the same queries

The dashboard and the alert rules should read the same expressions. When they drift apart, the alert fires and the dashboard shows nothing wrong, which is how teams learn to ignore alerts.

Two rules keep the channel readable. First, alert on symptoms, not on every underlying cause: page on realized or estimated performance and on prediction drift, and route feature-level drift to a low-urgency channel where it acts as supporting evidence during an investigation. Second, use burn-rate style multi-window conditions for the serving row rather than a single instantaneous threshold, following the SRE Workbook’s alerting-on-SLOs pattern, so a thirty-second blip does not page anyone at 3am. The routing, severity, and suppression design specific to drift is worked through in alerting for ML model drift.

Every alert should link back to the dashboard with the model-version and time-range variables pre-filled. An alert that requires the responder to reconstruct the query is an alert that gets acknowledged and closed without investigation.

Where the numbers come from

Three different backends feed this dashboard, and trying to force them into one is a recurring mistake.

• Time-series database (Prometheus or equivalent) for row 1 and for the computed drift metrics from row 3. Anything that is a number over time with low cardinality.
• Columnar store of prediction records for rows 2 and 3, joined to labels as they arrive. This is where cohort slicing happens, and it needs a query engine, not a metrics store.
• Trace backend for row 4 and for every drill-down. Cost and token attributes live on spans, and the panel is an aggregation over them. What to put on those spans is covered in end-to-end tracing for LLM applications, and the Trace Span Designer will generate the attribute set for a given pipeline shape.

The scheduled job that computes drift statistics and writes them as metrics is the piece that turns a data warehouse into a dashboard. Whether that job is Evidently, NannyML, or fifty lines of pandas is a smaller decision than it appears; the open-source ML observability stack covers how those components fit together.

Add a freshness panel for the dashboard itself

The failure mode nobody plans for: the prediction-label join stalls, the accuracy panel keeps rendering the last computed value, and the dashboard reports health for a week while the pipeline behind it is dead. Put a small panel in the header showing the age of the most recent computed metric for each row, and alert on it. A stale green dashboard is worse than no dashboard, because it actively suppresses investigation.

The same applies to the drift job. If the scheduled computation fails silently, the PSI panel simply stops updating, and a flat line reads as stability.

Link dashboard panels to experiment runs and traces

A release comparison needs a stable connection to the experiment that produced the deployed artifact. MLflow Tracking records parameters, metrics, and artifacts by run. Keep the equivalent run identifier from your chosen tracker in release metadata, alongside the artifact version and validation dataset identifier. The W&B vs MLflow vs Comet comparison covers how to choose that tracking workflow.

Use this suggested evidence map when designing panel navigation:

Panel	Link destination	Context to preserve
Quality by model version	Release record, then experiment run	Artifact version and evaluation dataset
Cohort performance	Prediction-label query	Cohort, prediction window, label cutoff
Route latency or token usage	Trace search	Route, model version, time range
Online evaluation score	Scored request records	Evaluator version, score definition, sample window

Grafana dashboard links can preserve time ranges and template variables. Data links can also use series labels and row fields. Configure each destination’s accepted parameters explicitly; a link that opens an unfiltered overview loses the incident context.

Show the number of labeled predictions beside a quality metric and distinguish missing data from a passing result. For generated answers, online evaluation supplies the scored sample; how to monitor LLMs in production connects those scores with serving telemetry. Keep an evaluator change visible alongside model and prompt releases so a scoring change is not silently attributed to the model.

Dashboard questions

Can an experiment dashboard replace the production view?

An experiment view compares recorded runs. A production view also needs live serving windows, label coverage, and request evidence. Link the two using release metadata; do not present a validation-set score as current production accuracy.

Should a drift panel trigger an automatic retrain?

Use it to investigate. A distribution change alone does not establish performance loss. Follow the model drift detection workflow to check available outcomes and data quality before choosing a response.

Review cadence

Dashboards rot. A quarterly pass with two questions per panel keeps this one honest: has this panel changed a decision in the last quarter, and is its threshold still the one that would have caught the last incident? Panels that fail the first question get deleted, thresholds that fail the second get rewritten. A dashboard that only ever grows is a dashboard on its way to being ignored.

For the layer model that this dashboard is an implementation of, and what to instrument before building any of it, start with ML observability: layers, signals, and architecture.

See also

• Debugging model accuracy drops in production
• Model drift detection: catching performance decay early
• How to monitor an LLM in production
• Weights & Biases vs MLflow vs Comet: choosing by constraint
• Alerting for ML model drift: a practical setup