Wiljan Cools's Linkedin Analytics

How do you monitor data quality once your model has been deployed? Maintaining the quality of data in production just as it was in training is not easy. Here’s a quick toolbox for tackling post-deployment data quality issues: 1️⃣ Univariate Drift Detection: Use methods like the L-Infinity Method or the Wasserstein Method to detect shifts in individual features. These methods, sensitive to outliers, can also help highlight extreme values in the data distribution. 2️⃣ Data Quality Checks: Identify Unseen Values or flag Missing Values to track data inconsistencies that might cause model failures. 3️⃣ Summary Statistics: Monitor Averages, Medians, and Standard Deviations to capture subtle changes in data trends over time. Each tool plays a role in understanding where data is drifting or degrading. Together, they form a robust framework to ensure your models are informed by quality inputs. Are you monitoring data quality post-deployment?

Are you monitoring your ML models the right way? Most companies monitor ML models by measuring realized performance and checking for data drift. This makes sense when labeled data is available, but what happens when it's not? or when data drift doesn't actually impact performance? Our ML workflow is about performance. So our monitoring workflow should also be about performance right? Here’s a simple way: •Estimate model performance ahead of time: Stop waiting for ground truth. Probabilistic algorithms like CBPE and DLE can estimate performance even without new labels. You can use custom metrics to interpret performance correctly. •Investigate only when performance drops: Drift detection should help you understand why a performance drop happens, not serve as an early warning system. •Context is key to solving issues: Once you identify the cause of a performance issue, tailor your solution to the context. There’s no one-size-fits-all fix.

Imagine if you see a performance drop will model retraining fix the issue? The answer is NO If the data drifted to a region where it’s harder to make correct predictions (like close to the class boundary), but the concept learned by the model is still accurate (so no concept drift), retaining will not fix the issue, as the model will learn the same pattern. If the root cause is data quality, retraining will only exacerbate the problem, as the model will learn a wrong pattern on bad quality data. The answer is YES If concept drift is why the model performance dropped, retraining is likely to resolve the issue fully. The presence of concept drift is the best trigger for model retraining, even better than just monitoring performance.

As a data scientist, you’ve likely encountered situations where existing ml metrics didn’t quite capture the nuances of your business KPIs. You needed a custom metric that could tell a more accurate story, but you didn't know how to get started in building and monitoring it. With NannyML Cloud, you can now easily define personalized metrics, monitor them, and estimate their impact—all in one place. For example, you might prioritize an F2 score instead of a standard F1 for cases where recall matters more than precision. What metrics would you create to better reflect your model's performance? Share your ideas in the comments below!

If the model stops working, the only issue is its unusability, right? But it can actually pose great risks to companies and institutions when the inevitable happens: consumers’ behavior changes, investments have been made, interests and inflation have been raised and liquidity has… evaporated into thin air. During the pandemic, Silicon Valley Bank invested heavily in debt security while interest rates were low assuming they would remain low, but after 2021 most of the tech sector was trying to get back on their feet post-lockdowns and, generally, an economic recession. It was not a great omen that most of SVB’s customers were, in fact, from the tech sector. And when most of your clients need cash all at once, chaos ensues. All those low interest rates investments had now turned into high interest, and the low-yield treasury bonds that were supposed to pay interest had lost all their profit due to inflation. Hence, unrealized losses. Lots of them. Which created even more unsteadiness among their clients. Which ultimately led to the bank’s downfall. According to some, SVB’s fallout was the cause of the employment of “bad models” while assessing its own risks. Could this have been avoided? Yes. By monitoring ml models.

How does data drift affect machine learning model performance? We had published a comprehensive blog answering this exact question long ago But for me, it is still one of the most complete overviews of the subject I've seen I recommend giving it a read if you want to fully understand: ▪How covariate shift can impact model performance and how it can sometimes actually improve it ▪How concept drift impacts performance and how to calculate the exact integrals that allow us to quantify this impact ▪How concept drift and covariate shift can interact to make problems even worse than they seem at first glance Check the blog:

MLOps V/S Post Deployment Data Science MLOps is about automating the building, deploying, and operational monitoring of your machine learning models Post-Deployment Data Science is about model performance monitoring, root cause analysis, and issue resolution There are parallels between the two but from a completely different perspective with monitoring, a data scientist would be responsible for model performance metrics like ROC AUC, F1, or RMSE or any other custom business value metric might have ML engineers would be responsible for monitoring latency, uptime, and infrastructure costs For the past few years, we've been focusing on MLOps It's now time for data scientists to go post-deployment

Data Scientist in the past: ⏩ Focused primarily on building and deploying models ⏩ Success measured mainly by deployment metrics. Data Scientist now: ⏩ Continuously monitors and maintains models for optimal performance. ⏩ Measures success through business value and impact on decision-making.

When a credit risk model is in production, its performance will degrade over time. In one case, several features that exhibited univariate drift were removed in an attempt to stabilize the model. However, this did not reduce drift or improve performance; those features were critical to the model’s decision-making. The issue was more complex than univariate drift alone could explain. A different approach was needed. Instead of looking at features in isolation, PCA Reconstruction Error was used to measure multivariate drift. Surprisingly, a single feature that appeared stable and showed no univariate drift was the key driver of multivariate drift. Removing it significantly improved model performance, particularly the GINI coefficient. This highlights the importance of considering multivariate interactions when monitoring models. If you're interested in learning more about advanced monitoring techniques, check out our webinar: https://lnkd.in/e4NJaCfC

Never monitor data drift alone. We care about model performance, so that’s what we should be tracking. And with methods to estimate performance metrics, the old excuse of “we don’t have ground truth yet” is no longer valid. Check out the plot below. One of the main model's features shows data drift alerts, but performance wasn't affected. Not. Every. Drift. Affects. Model. Performance. In his blog, Santi replicates a paper and properly applies a performance-centric monitoring workflow. Read it in the comments

Maintaining an ML model is a full-time job. There’s a lot to take care of: 🔷 Monitoring for data drifts and pipeline issues. 🔷Debugging errors when model performance drops. 🔷Dealing with constant alerts that make it hard to know what’s important. Managing all of this manually can quickly get out of hand. That’s where cloud solutions come in. They help centralize monitoring, automate root cause analysis, and reduce the noise from irrelevant alerts. Instead of wasting time firefighting, you can focus on improving your models and delivering better results. How do you manage ML monitoring today?