Debugging Model Accuracy Drops in Production

The weekly report shows accuracy down four points. The room’s first instinct is to retrain, and retraining is the wrong move roughly as often as it is the right one. It is expensive, it takes days, and when the cause was a null-filled feature or a preprocessing mismatch, the new model inherits the same defect and the number comes back down a week later.

A drop is a symptom. There are five plausible causes, they are not equally likely, and they are not equally expensive to rule out. This is the order to test them in, what evidence disqualifies each one, and what has to be instrumented in advance for any of it to be possible.

Step 0: is the drop real?

Before anything else, establish that the number moved because the model got worse. Measurement artifacts produce more false alarms than genuine degradation does, and each of these is cheap to check.

• Label lag. If labels arrive on a delay, the most recent window is partially labeled, and the labels that arrive first are usually a biased subset. Fraud outcomes and churn confirmations skew toward the fast, obvious cases. Always compare fully-matured windows against fully-matured windows.
• A broken join. The metric is computed on a prediction-to-label join. If the join key changed, or a schema migration renamed a column, the join silently drops rows, and it rarely drops them at random.
• Sample size. A four-point move on a segment of 300 rows may be noise. Put a confidence interval on the metric before treating a change as a signal.
• Mix shift. More on this below, because it is the cause that survives every naive check.

If any of those explains the move, stop. The model is fine and the measurement pipeline needs the fix.

Step 0.5: aggregate metrics hide mix shift

Aggregate accuracy is a weighted average over segments, and the weights move. A model can hold its per-segment accuracy exactly and still show an aggregate drop, purely because traffic shifted toward a segment it was always weaker on. Marketing launches a campaign, a new region ramps, a mobile client ships, and the mix changes overnight with no change in model behavior at all.

The decomposition worth running before anything else: for each segment, record last period’s volume share, this period’s volume share, and the per-segment metric in both. If the per-segment numbers are flat and only the weights moved, the model did not degrade. The correct response is a conversation about whether the new mix is acceptable, not a retrain.

This is the reverse case as well. An aggregate that looks flat can conceal a segment that fell off a cliff, offset by growth in an easy segment. That is why the monitoring dashboard spec puts performance-by-cohort next to aggregate performance rather than behind a drill-down.

The triage order

Cheapest and most likely first. Each step below has an evidence test that either implicates it or clears it, so the investigation converges instead of wandering.

Order	Hypothesis	First check	Clears it if
1	Input pipeline broke	Null rate, freshness age, cardinality per feature	All inputs steady and fresh at the drop time
2	Something was deployed	Model version, feature-transform version, dependency versions	No version change in the window
3	Training-serving skew	Offline vs online feature values for the same entity	Values match to floating-point tolerance
4	One segment, not the population	Metric sliced by cohort, region, client, device	Every segment moved by a similar amount
5	Genuine distribution shift	Feature and prediction drift against a pinned reference	Inputs and outputs statistically unchanged

1. The input pipeline

Most “model degradation” is data degradation. An upstream service starts returning nulls for a field, an ETL job silently truncates, a third-party enrichment API rate-limits and the client fills the gap with a default. The model keeps scoring, because a missing value is usually imputed rather than rejected, and the imputed value is quietly wrong.

The check is a per-feature null rate, a freshness age, and a value-range assertion, all timestamped so they can be aligned against the drop. Note the direction of the fix here: this is a data-platform incident, not a modeling one, and paging the ML team for it wastes the first hour of every investigation of this kind.

2. Something was deployed

Correlate the drop against every deployment in the window, and not only model deployments. The feature transformation code, the serving container’s library versions, and the upstream producers of every input all count. A minor version bump in a tokenizer or a scaler is enough to move a distribution.

This is why the model version and the reference-window version belong in the dashboard header as filters rather than inside a panel. Comparing a window that straddles a deploy against one that does not is the single most common way an investigation reaches the wrong conclusion.

3. Training-serving skew

Skew is when the features a model receives at serving time differ from the ones it saw in training, for the same logical input. Google’s Rules of Machine Learning treats this as one of the central production hazards, and its recommended defense is still the right one: log the features exactly as they arrived at serving time, then score the model on those logged features offline and compare.

The mechanism is usually duplicated logic. A transformation is implemented once in the training pipeline and again in the serving path, and they diverge on an edge case: a different rounding rule, a category the online encoder has never seen, a timezone. The 2015 NeurIPS paper on hidden technical debt in ML systems named this class of problem, and the structural fix has not changed: one implementation, shared by both paths.

Skew tends to produce a step change rather than a slope, which is a useful discriminator against genuine drift.

4. A segment, not the population

Slice before concluding anything global. Run the metric by cohort, by region, by client version, by device class, and by any feature with a small number of high-volume values. A global retrain aimed at a defect that lives in one segment usually trades accuracy elsewhere to fix it, which is a bad deal that shows up as a second incident later.

5. Genuine distribution shift

Only after the first four have been cleared is drift the leading hypothesis, and even then the distinction matters. Covariate shift means the inputs moved; concept drift means the relationship between inputs and the target moved, and the second one is the one retraining actually fixes. The signals that separate them, and the estimators that flag decay before labels arrive, are worked through in model drift detection.

Label-free performance estimators are particularly useful in this step, because they say whether the input shift is one the model is sensitive to. A large PSI move on a feature the model barely weights is noise; the NannyML performance-estimation approach is designed to answer that question directly rather than leaving the team to guess from a drift chart.

Debugging LLM systems: the same order, different probes

For a generative system the drop is usually a quality-eval score rather than accuracy, but the triage order survives with substitutions.

• The input pipeline becomes the retrieval layer. A stale index, an embedding-model change, or a recall regression looks exactly like the model getting worse. That layer is covered in embedding and vector-store observability.
• “Something was deployed” includes prompt-template edits and provider-side model updates, which happen without a deployment on your side. Pin and log the model identifier on every span.
• Skew becomes the eval-prod gap: the offline suite and the production traffic distribution diverge, which is a structural problem rather than a measurement error. See closing the eval-prod gap.

The prerequisite for all three is a trace with enough on it. If retrieval results, prompt version, model identifier, and token counts are not on the span, none of these checks can be run after the fact. What belongs in a span covers the schema, and the Trace Span Designer will generate the attribute set for a given pipeline shape.

What has to exist before the incident

Every check above is a query against data that either was collected or was not. Retrofitting it during an incident is not possible, so the following are prerequisites rather than improvements.

• Prediction records. Every scored request, stored with its features as they arrived, the model version, and a stable identifier that labels can be joined to later.
• A pinned reference distribution. Versioned, tied to the training set, and never a rolling window. A moving baseline makes gradual drift invisible by construction.
• Segment keys on the record. Cohort, region, and client belong on the row at write time. They cannot be reconstructed afterward.
• Deployment events as annotations. A timeline of model, code, and dependency changes that the dashboard can overlay.

None of the four is an alert rule, which is the point: a dashboard tells you the number moved, and only the underlying records tell you why. That split is the subject of ML observability vs monitoring. The architecture that produces all four is set out in ML observability: layers, signals, and architecture, and the alert routing that should trigger this playbook is in alerting for ML model drift. Symptom-level alerting, in the sense the SRE Workbook uses it, is what keeps this triage rare enough to be worth doing carefully.

Close the loop in writing

Record the cause, the evidence that identified it, and the check that would have caught it sooner. Most teams find the same two or three causes recurring, and each recurrence is an argument for a new panel or a new assertion rather than a faster investigator. A drop that gets diagnosed in twenty minutes because someone added a null-rate alert after the last one is the entire return on this work.

See also

• ML observability vs monitoring: what actually differs
• Model drift detection: catching performance decay early
• ML model monitoring dashboard: what to put on it
• How to monitor an LLM in production
• The open-source ML observability stack: Evidently to Phoenix