NannyML

NannyML is an Open Source Python library to detect silent model failure, estimate the performance of ML model even without ground truth, and robustly identify data drift.

Co-Founder and CEO

Prophecy Labs

Founder

Data Solutions. Elegantly Crafted.

Prophecy Labs helps companies automate complex work, optimize business processes, and predict events that impact your business.

Let our passion, be your edge.

Managing Partner

Essent.be

Developing a data platform in Scala/Spark to deal with large scale data and
deploying large scale machine learning models.

Big Data Engineer

SCK-CEN

This project was to help prepare for long term space travel namely a long term lunar outpost within the following decade. It involved working with gene expression data to help determine the effects of long term isolation and harsh condition on the human immune system. Data from workers of the Concordia platform in antarctica was used. Technical details of the R code can be found at the link below.

Research Bioinformatician 

Building an open source library to estimate the performance of deployed machine learning models in the absence of ground truth.

I love talking about: machine learning, decision making, post deployment data science, performance estimation, and bunch of other stuff. 

currently learning about: causal inference 

Always open to have a chat 🙂

Leave me anonymous feedback -> https://www.admonymous.co/hakim_elakhrass

Every data science project you start needs to start with a business case


most data science projects worth doing are complex

its not good enough to have a highly performant model

to deploy the project in production you are most likely going to have to change workflows across multiple teams and get buy-in from more people than just your boss

after all the changes and complexity the project might not even be worth it, given the cost of deploying and maintaining it

the only way to be sure that your projects get into production and bring ROI is to start with a business case and bring all the stakeholders around the table

🥸Hakim Elakhrass

post-deployment data science | OSS | co-founder @ nannyML

Probabilistic forecasting is a key data science use case.


That’s why we’re sponsoring multiple prizes for the first-of-its-kind forecasting competition on Kaggle.

The prizes are:
▪ A 6-month license of NannyML cloud on the cloud of your choice.
▪ Two copies of the forecasting Bible: Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos.

Check out more the details here about the prize:
https://bit.ly/3JpDQDi

-------
this post was mostly plagiarised from Wojtek Kuberski

graphical user interface, text, application, email

How do you know what caused your model performance to change? 
 
 
if you see that your model performance has changed in production, the first question that will come to mind is, why? 
 
There are 5 things to check as a data scientist in a root cause analysis workflow on the batch of data where your model performance has changed 
 
1. Data Quality 
 
One of Most common causes for model failure. More missing values than expected? 0 and 1's flipping (yes this happens in production 😳). Outside temp. at 200 degrees? 
 
3. Univariate drift 
 
i throw a lot of shade on univariate drift detection techniques but they are great for root cause analysis. What features distribution significantly changed in the batch where performance changed? 
 
2. Multivariate Drift 
 
univariate techniques still can't detect changes in correlation or general structure of the data obviously (gotta throw the shade 😏). Relationships between distributions can change even when the individual distribution has not changed significantly.  
 
4. Concept drift (when labels are available) 
 
when labels are available, it is actually possible to detect it and measure its impact on performance. (also pro tip, if thats the only issue detected you can just retrain on the latest data). 
 
5. Check with the responsible business unit 
 
finally you potentially need to check with the business unit where the data is generated or predictions used to understand further. Maybe the marketing department ran a new campaign that you didn't know about, and caused different types of customers to buy your product, messing your models up.  
 
Am I missing anything? What do your Root Cause analysis workflows look like?

Isn't a data science pilot the same as a proof of concept (POC)?


No. A POC is when you validate the feasability from a data science perspective.
 
You check:
1. Do I have the data (also in production)
2. Can I build a model that solves the problem better than the benchmark

Once you have a model that you expect to give a positive ROI a pilot validates in the real world. 

You deploy that model in a limited fashion, but you actually make decisions based on the predictions. I.e. a inventory management model deploying into a limited number of stores on a limited number of products.

You check:
1. Does the model actually work in the real world
2. Is there is a positive ROI when we take into account the complexities of the real world.

Of course you do all the homework possible ahead of writing any code to build a business case and get all the stakeholders onboard. POC's and then pilots are to validate the assumptions of your business case. 

At the end of the pilot you get a go/no-go on whether you should scale your use case. 

Do you run real world pilots of you data science use cases?

Machine learning models are not the only way data scientists can predict the future


In the next decade I believe more data scientists will be starting startups. We are uniquely positioned to understand todays world. 

Good data scientists understand the intersection of data, technology, and business. We can understand what problems are valuable to solve with upcoming and current technology, before anyone else.

In the coming weeks i'll be releasing more videos on the intersection of data science and entrepreneurship based on my personal experience over the past 5 year. 

Check out the full video in the comment section.

Do you think data scientists make good startup founders?

Startup's and PhD's share a dark secret


The expected returns are extremely low. Social media and society in general glamourise both extensively. 

Taking into account opportunity cost, your expected life time earnings when you do a startup or PhD are going to be lower than if you don't. 

Especially as a data scientist.  

I think it is the duty of founders and academics to be more upfront about this. 

This doesn't mean I think you shouldn't do a startup (PhD's are another story). It's just the reason behind doing them should not be monetary. 

Startup's make a lot more sense than most PhD's imo (check the rest of the video in the comments to hear more).

What are your thoughts?

HOW DOES THE BODY OF A SALAMANDER KNOW WHEN TO STOP RE-GROWING AN ARM?


Where is that information stored?

"Complex agents often consist of components that are themselves competent problem solvers within their own systems"

It's been 10 years since I dove deep into Bioinformatics and evolutionary biology. Given all this talk about AGI it seems to be more relevant than ever.

In this paper by Michael Levin, he proposes the TAME framework for universally classifying intelligence regardless of origin. 

A dog and a GPT can be put on the same scale defined by the time and space of the goals that they can persue. 

I have never read a paper that redefined my understanding of a fundamental concept. This is one of them. 

The keys to true AGI lie in unlocking all the different levels of intelligence that nature has discovered.  

Intelligence is diverse. 

This has huge implications.

Recommend giving it a read. Link in the comments.

3 reasons you need to measure the impact of data drift on your model performance

1. Singe metric to communicate and monitor

using techniques like CBPE and DLE you can get an estimation of any model performance metric in production (F1, MAE etc), even when ground truth is not available. These techniques are actually quantifying the expected impact of data drift on your model performance. 

2. Trigger actions based on that metric

because you have a single metric, you can set simple thresholds around it and trigger workflows when your model performance drops below them.

for example exploring what the issue was or automated retraining (here be dragons though). 

3. Significantly reduce false alarms

since you aren't doing 100's of statistical tests like KS or JS, this will only alert you when there is something to actually be concerned about. Reducing false alarms significantly. You can still use these statistical tests to find the root cause of the performance change.

all of this saves you a ton of time and makes communicating about your model performance to your team mates and other stakeholders significantly easier

model performance in production is serious stuff. bad things happen if we don't do our job as data scientists. 

Let me know in the comments what your monitoring workflows look like.

What type of data science product should you build?


What is a data science product even?

Does it really matter? Not really, but chances are if you are reading this, and watching my video you are a data scientist. 

If you want to build a product you are going to start from what you know. It might be your unfair advantage.

At the end of the day the only thing that matters is building something that adds tremendous value. Including or excluding data science. 

It will prolly take you a decade though, so you better love it.

I discuss the different types of products and who is best fit to build what in the full video. Check it out in the comments below.

What type of product would you build?

Reminder to not use SMOTE


Some people complain that this AI music has no soul.

I think all the songs I've generated are full of soul 

but that might just be because I'm asking it to sing about data and that's part of its soul 


what do you think about AI music?

monitoring the performance of your model is trivial right?  
 
Wait until you get ground truth, calculate performance, sleep well. 
 
Unfortunately the real world is a lot more complex. Most use cases have a delay in ground truth.  
 
 In the time between your first batch of predictions and your ground truth becomes available the world could have changed dramatically.  
 
Covariate shift, concept drift or data quality changes can have caused your model to fail. All the while you are still using it to make new predictions. 
 
Using estimated model performance in nannyML cloud, you can estimate model performance before the ground truth comes in. If it drops significantly you can trigger workflows to resolve the issue before it impacts business. 
 
You can trial it out for free on Azure and AWS.  -> https://www.nannyml.com/

Managed Applications are a miracle of modern technology


they allow you deploy software within your VPC. 

no data leaves your system. 

they can follow Microsoft standard TOC's. 

they get billed via Azure. 

it always surprised me how few teams know about managed applications. 

completely automated, and deployed within a few minutes, 

every IT department we pitched them too, have loved them. Most didn't know they existed. 

They also often have free trials. If you haven't tried them out, i suggest you do, then can make your life a lot easier.

Any favourite managed applications? Let me know in the comments.

data scientists need to be performance maxxing their models

Nice proposal of using CBPE or DLE for feature selection.

Thanks a lot Ilia Ekhlakov for sharing. One of the cool things of working on fundamentals is the applications can go beyond what you intended. 

We've seen applications in model selection as well. 

On a separate note, I have heard of some people using PCA-Reconstruction error directly for predictive maintenance, which is cool.

Senior Applied Data Scientist @ Wrike | B2B SaaS | Revenue Strategy & Ops | MSc in Physics | Lifetime learner

Great thanks to #nannyml for their amazing algorithms!

It turns out they have algorithms that can accurately evaluate performance of both classification (M-CBPE) and regression (DLE) models on new data, even before the target values are available. I'll add links with detailed descriptions in the comments. While here I'd like to discuss why is this so great.

The importance of having such algorithms for monitoring models in production has been brilliantly explained by #nannyml themselves. But this is only the beginning of the list of their possible successful use cases. Let me share my proposals on a few more of them.

In one of my previous posts, I touched upon the problem of feature selection when there is a long time gap between the freshest data for which a target is available and the data coming in for inference. Traditional feature selection methods based on the impact of adding/dropping a feature on validation metrics are not very useful if the validation data is significantly different from the inference data. That time my suggestion was to use the Population Stability Index (PSI) to detect features with the most significant distribution shift and to take a closer look at them. But how exactly do this? 

Imagine that you couldn't find a transformation to stabilise the distribution of a problematic feature without noticeable worsening its metrics on validation. The answer to the question of whether to drop it is hard to formalise, thus it requires experience, a good understanding of prediction patterns, and experimentation. Of course, the probability to make a wrong decision remains high.

With M-CBPE and DLE, such a decision can be made with maximum justification, practically eliminating the probability of errors: based on the change in selected performance metrics values expected on inference.

In addition, these algorithms allow us to evaluate model quality metrics in situations where the difference between training and prediction data is expected from the outset. For example, if representativeness was lost during dirty data cleaning, or when creating the training dataset by matching.

#Kudos to #nannyml and thank you for your contribution to #datascience!

UPD: please, keep in mind that both methods cannot cope with the situation when the concept shift takes noticeable place. So, we are waiting for new brilliant solutions from #nannyml

#machinelearning #modelmonitoring #featureselection #modelevaluation

The nannyML groove


Normally I am skeptical about 'groundbreaking' genAI releases. They usually come out with very short demo clips and allow only limited number of users.

Yesterday the world changed forever. Udio released a music generator, open to the public. I saw it within an hour of releasing and it wouldn't let me generate anything, so I was skeptical. 

Today it actually worked. I was able to generate a jingle about nannyML in less than 5 minutes. 

We live in interesting times. Check comments for other versions.

Taplio

🥸Hakim Elakhrass's Linkedin Analytics

🥸Hakim Elakhrass

🥸Hakim Elakhrass's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}🥸Hakim Elakhrass's Linkedin Analytics

🥸Hakim Elakhrass

🥸Hakim Elakhrass's Best Posts .css-qqfgvy{font-size:var(--chakra-fontSizes-md);}(last 30 days)

Want to drive more opportunities from LinkedIn?

🥸Hakim Elakhrass's Linkedin Analytics

🥸Hakim Elakhrass's Best Posts (last 30 days)