The MLOps Gap: Why Model Quality and System Quality Are Different Problems

Machine learning research optimises for model quality. ML engineering in production optimises for system quality. These are not the same problem, and confusing them is one of the most expensive mistakes a team can make.

What Model Quality Measures

Accuracy, F1, BLEU, perplexity, human preference ratings. These metrics measure how well a model performs on a held-out test set under controlled conditions. They're necessary but not sufficient.

What System Quality Requires

Latency SLAs. Throughput under concurrent load. Graceful degradation when a dependency is unavailable. Meaningful error rates. Drift detection when production distribution diverges from training data. Cost per inference at scale.

None of these show up in a model card.

The Three Gaps

Most teams hit the same three gaps:

1. The observability gap — no instrumentation to detect when model behaviour is degrading in production, only user complaints or business metric drops.

2. The feedback gap — no pipeline to capture production examples and route them back into training or evaluation sets. The model can't learn from its mistakes.

3. The governance gap — no documented process for model rollbacks, A/B testing new versions, or incident response when a model causes harm.

The Fix

Treat your ML system like a production service. Write runbooks. Define SLOs. Build shadow evaluation pipelines. Make deployment a boring, repeatable process rather than a high-stakes event. This is not glamorous work. But it's what separates systems that scale from systems that become liabilities.