NannyML

NannyML is an Open Source Python library to detect silent model failure, estimate the performance of ML model even without ground truth, and robustly identify data drift.

Co-Founder and CEO

Prophecy Labs

Founder

Data Solutions. Elegantly Crafted.

Prophecy Labs helps companies automate complex work, optimize business processes, and predict events that impact your business.

Let our passion, be your edge.

Managing Partner

Essent.be

Developing a data platform in Scala/Spark to deal with large scale data and
deploying large scale machine learning models.

Big Data Engineer

SCK-CEN

This project was to help prepare for long term space travel namely a long term lunar outpost within the following decade. It involved working with gene expression data to help determine the effects of long term isolation and harsh condition on the human immune system. Data from workers of the Concordia platform in antarctica was used. Technical details of the R code can be found at the link below.

Research Bioinformatician 

Building an open source library to estimate the performance of deployed machine learning models in the absence of ground truth.

I love talking about: machine learning, decision making, post deployment data science, performance estimation, and bunch of other stuff. 

currently learning about: causal inference 

Always open to have a chat 🙂

Leave me anonymous feedback -> https://www.admonymous.co/hakim_elakhrass

We are going to NeurIPs 2025!


Super proud of the team.

🥸Hakim Elakhrass

building Soda to monitor the worlds data and decisions

Research Scientist at SODA / NannyML, PhD

You can accurately estimate classification model performance without access to labels under covariate shift.

You heard it right - WITHOUT LABELS. To find out exactly how, check our paper* accepted at NeurIPS 2025 where we introduce Probabilistic Adaptive Performance Estimaton (PAPE). It was a long journey in the great company of Juhani Kivimäki, Wojtek Kuberski, Nikolaos Perrakis, and Jukka K. Nurminen.

It started with a simple yet clever idea that the higher the confidence a classifier has in its prediction, the more likely it is to be correct ([A], Dan Hendrycks). Intuitively, if your binary classifier predicts label 1 with y_pred_proba of 0.9 it is more likely to be correct than with y_pred_proba of 0.6. It was then turned into a method called Average Confidence (AC) - where you just estimate accuracy of a classifier by taking average of confidence scores on a sample. That method later became a naive benchmark for supposedly superior approaches. Yet we recognized that it was undervalued - we noticed it is an instance of another broad and simple principle: if you have access to the true conditional probability P(Y|X), you can estimate any performance metric. Sticking to accuracy for simplicity: if you have n observations with y_pred_proba of 0.9 and a binary prediction of 1, and if this confidence is calibrated (represents probability), 90% of your samples will be correctly classified. So your accuracy will be 0.9. It is possible to estimate also other metrics and to even get their probability distributions (using Poisson binomial distribution) - hence also the intervals. The key problem is that calibration degrades under certain distribution shifts so it is not enough to calibrate on your held out set once - calibration should be adaptive to the distribution of the sample you want to estimate performance for. This is what PAPE does.

We described the story above in 3 papers, all published in 2025:

- in [1] we analyze the AC method (accepted at JAIR),
- in [2] we introduce Confidence-Based Performance Estimaton (CBPE) capable of estimating metrics other than accuracy - the one you know from NannyML (acq. by Soda Data Quality) (accepted at ACML),
- in [3]* we introduce PAPE by ensuring CBPE maintains calibration under covariate shift (accepted at NeurIPS).

Text me if you would like to find out more, collaborate, discuss performance estimation topics, talk about your use case, or NannyML usage.

On a personal note, the best part of this research wasn’t just the publications - it was meeting Juhani Kivimäki, who, as it turned out, had been working on the very same problem all along, somewhere in the dark and cold of Finland 😊. This is probably not our last word on unsupervised performance estimation.

Taplio

🥸Hakim Elakhrass's Linkedin Analytics

Performance Score

Who is engaging with 🥸Hakim

's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?