Paul Iusztin

Decoding ML

Battle-tested content to help you design, code, and deploy production-grade ML & MLOps systems.

Founder

Remote Work

💼 Senior machine learning engineer, contractor, consultant, and freelancer with 𝟔+ 𝐲𝐞𝐚𝐫𝐬 𝐨𝐟 𝐞𝐱𝐩𝐞𝐫𝐢𝐞𝐧𝐜𝐞 offering scalable and modular machine learning solutions for businesses worldwide.

➞ I can design, implement, train, evaluate, and operate end-to-end ML pipelines for your business. I can scale up your ML systems by leveraging the latest MLOps technologies.

➞ I have spent 7+ years writing code in Python, but I also enjoy using Node.js for web development and C/C++ for optimizing various applications.

➞ My go-to machine learning frameworks are PyTorch, Keras, and Sklearn. Also, my favorite MLOps tech stack is AWS, W&B/MLFlow, and DVC.

🔥The world of AI is constantly changing, new technologies are born almost daily, and therefore, I am always open to new challenges.

💬 𝐂𝐨𝐧𝐭𝐚𝐜𝐭 𝐦𝐞, and let's see how I can help your business.

Senior Machine Learning Engineer • Freelancer • Contractor

Metaphysic.ai

🟡 Computer vision & deep learning workflow engine: orchestrate & execute

The client, a leading GenAI platform, faced difficulties scaling and breaking down its complex ML jobs. Their manual processes and CLI-based tooling added significant friction in serving their clients. Thus, the client required a more intuitive and scalable tool that could be used across the company to train and serve its ML models.

→ 𝗥𝗲𝗳𝗮𝗰𝘁𝗼𝗿𝗲𝗱 the ML worker architecture to follow DDD principles, making it more 𝗿𝗲𝗮𝗱𝗮𝗯𝗹𝗲, 𝗺𝗼𝗱𝘂𝗹𝗮𝗿, and 𝗲𝘅𝘁𝗲𝗻𝘀𝗶𝗯𝗹𝗲.

→ 𝗗𝗲𝗰𝗿𝗲𝗮𝘀𝗲𝗱 the 𝗹𝗮𝘁𝗲𝗻𝗰𝘆 of the data curation DL models 𝗯𝘆 𝟴𝟮.𝟳%

→ 𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱 their clustering algorithm to accurately run on 𝗱𝗮𝘁𝗮𝘀𝗲𝘁𝘀 >𝟭𝟬𝗸 𝗶𝗺𝗮𝗴𝗲𝘀 and 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗰𝗮𝗹𝗹𝘆 𝗱𝗲𝘁𝗲𝗰𝘁 the number of 𝗰𝗹𝘂𝘀𝘁𝗲𝗿𝘀 and 𝗼𝘂𝘁𝗹𝗶𝗲𝗿𝘀.

Senior Machine Learning Engineer

⚫️ Scalable ML batch pipeline for continuous training & inference

The client, a leading GenAI platform, struggled with manual training and calling their ML models. As a tedious and unscalable process, they needed to automate their ML processes completely and integrate them with their current web platform.

→ 𝗗𝗲𝘀𝗶𝗴𝗻𝗲𝗱 and 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗲𝗱 a 𝗺𝗼𝗱𝘂𝗹𝗮𝗿 𝗣𝘆𝘁𝗵𝗼𝗻 𝗽𝗮𝗰𝗸𝗮𝗴𝗲 that orchestrates the ML steps into 3 fully automated batch pipelines: feature, training, and inference, 𝗲𝗹𝗶𝗺𝗶𝗻𝗮𝘁𝗶𝗻𝗴 𝗮 𝘁𝗲𝗱𝗶𝗼𝘂𝘀 𝟮-𝗵𝗼𝘂𝗿 𝗺𝗮𝗻𝘂𝗮𝗹 𝗽𝗿𝗼𝗰𝗲𝘀𝘀/𝘃𝗶𝗱𝗲𝗼.

→ 𝗥𝗲𝗱𝘂𝗰𝗲𝗱 the 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝘁𝗶𝗺𝗲 of the batch ML pipeline 𝗯𝘆 𝟲𝟮.𝟱% while preserving the same accuracy and quality.

→ 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗲𝗱 𝗠𝗟𝗢𝗽𝘀 𝗯𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 such as IaC, CI/CD, monitoring, experiment tracking, and model registries, making the system reproducible, testable and trackable.

→ 𝗗𝗲𝗽𝗹𝗼𝘆𝗲𝗱 a scalable & cost-effective 𝗮𝘀𝘆𝗻𝗰 𝗯𝗮𝘁𝗰𝗵 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 on top of AWS ECS & SQS that scales in and out based on the number of jobs, 𝗿𝗲𝗱𝘂𝗰𝗶𝗻𝗴 their 𝗔𝗪𝗦 𝗰𝗼𝘀𝘁𝘀 𝗯𝘆 𝟱𝟮%.

⚫️ Big data ingestion & archiving pipeline

The client wanted to expand its platform features by securely ingesting, preprocessing, and storing TB of data into their platform.

→ 𝐂𝐨𝐧𝐭𝐫𝐢𝐛𝐮𝐭𝐞𝐝 to the 𝐝𝐞𝐬𝐢𝐠𝐧 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 of a 𝐝𝐚𝐭𝐚 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐧𝐠 >𝟓 𝐓𝐁 of data in 𝐨𝐧𝐞 𝐠𝐨 in a reliable, fast, and secure way.

→ 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗲𝗱 a scalable 𝗱𝗮𝘁𝗮 𝗽𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 that supports multiple media types (images, videos, audio, 3D objects), archiving the original files on an affordable storage solution, 𝗿𝗲𝗱𝘂𝗰𝗶𝗻𝗴 𝗰𝗼𝘀𝘁𝘀 𝗯𝘆 𝘂𝗽 𝘁𝗼 𝟴𝟭%.

Towards Data Science

⚫️ ML and MLOps engineering blog

Helping machine learning engineers level up through hands-on practical articles and tutorials on designing and productionizing ML systems.

➞ Published 𝐓𝐡𝐞 𝐅𝐮𝐥𝐥 𝐒𝐭𝐚𝐜𝐤 𝟕-𝐒𝐭𝐞𝐩𝐬 𝐌𝐋𝐎𝐩𝐬 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 with 90k+ views on Medium and 600+ stars on GitHub. An MLE & MLOps course that teaches you how to design, build, and deploy an end-to-end ML batch system: https://github.com/iusztinpaul/energy-forecasting

Machine Learning Engineer / Technical Writer

CoreAI

⚫️ Real-time social media recommender system using machine learning & Kafka

The client, a social media start-up from Israel, needed a real-time dashboard and recommender system on top of their Kafka infrastructure using modern ML and MLOps solutions.

➞ 𝐋𝐞𝐝 the architecture system design of a modular real-time social media recommender system built on top of the client's Kafka nervous system that leverages streaming and ML technologies such as Kafka Streams, Kotlin, PyTorch, Python, and Docker.

➞ Quickly 𝐩𝐥𝐚𝐧𝐧𝐞𝐝 and 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐝 a scalable recommender system based on heuristics, picked through A/B testing, using Kafka Streams and Kotlin. It served as a vital entry point and baseline for future algorithms.

➞ 𝐃𝐞𝐬𝐢𝐠𝐧𝐞𝐝 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a dashboard that computes KPIs in 𝐫𝐞𝐚𝐥-𝐭𝐢𝐦𝐞 𝐟𝐨𝐫 𝟏𝟎𝟎𝐤+ 𝐮𝐬𝐞𝐫𝐬 on the client's Kafka infrastructure using ksqlDB, SQL, Elasticsearch, and Apache Superset.

Machine Learning Tech Lead

⚫️ coreControl - end-to-end MLOps infrastructure

The client, a machine learning service provider, required an end-to-end ML/MLOps infrastructure to scale their operations and save development resources across multiple AI projects.

➞ 𝐋𝐞𝐝 the 𝐝𝐞𝐬𝐢𝐠𝐧 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 of coreControl by integrating the software with the client's products and collaborating with other ML engineers, developers, and graphic designers.

➞ 𝐑𝐞𝐝𝐮𝐜𝐞𝐝 the 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 𝐢𝐦𝐞 across the client's machine learning projects 𝐛𝐲 𝟑𝟎% using coreControl.

➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 the data backbone of coreControl that syncs the data between multiple MLOps tools and exposes them to a UI through a RESTful API using Python, FastAPI, PostgreSQL, Docker, and AWS.

➞ 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐝 a Python package that wraps various MLOps tools that standardizes the data scientists' work and centralizes all the ML artifacts to easily be analyzed in a single place using Python, DVC, and ClearML.

⚫️ Prototyping AI solutions for Demand Forecasting, Churn Prediction, and Customer Segmentation

The client, a machine learning start-up, wanted to expand their services portfolio by prototyping and gaining knowledge in various AI fields that leverage structured retail data.

➞ 𝐋𝐞𝐝 𝐚 𝐭𝐞𝐚𝐦 𝐨𝐟 𝟐 to develop an ML clustering solution that aggregates data from multiple sources and finds customer segments and insights for new potential clients.

➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a prototype for churn and demand forecasting used as a proof of concept to close deals with various clients.

➞ 𝐒𝐚𝐯𝐞𝐝 𝟒𝟎% of 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬 by building a reusable Python ML framework that can quickly be tailored to custom problems based on structured data.

Politehnica University of Timisoara

⚫️ Teaching the foundations of 𝐚𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐢𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 laboratory.

➞ Supervised and unsupervised learning
➞ Linear regression, SVMs, Decision trees
➞ Clustering, Dimensionality reduction
➞ Dense neural networks
➞ Convolutional neural networks
➞ Recurrent neural networks
➞ Transformers

Teaching Assistant

Everseen

🟡 Autonomous supply chain system using computer vision & deep learning

The client, a well-known player in the retail sector, needed to expand its deep learning and computer vision solutions by researching and building autonomous systems for its operations in the retail industry.

➞ 𝐓𝐫𝐚𝐢𝐧𝐞𝐝 a Mask-RCNN model for 2D object detection and segmentation on 100k proprietary retail images, with an 𝐦𝐀𝐏 𝐨𝐟 𝟖𝟓%.

➞ 𝐒𝐩𝐞𝐞𝐝 𝐮𝐩 the 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐭𝐢𝐦𝐞 of the Mask-RCNN model 𝐛𝐲 𝟓𝟎% by levering Docker, ONNX, and TensorRT to containerize and accelerate the models.

➞ 𝐓𝐫𝐚𝐢𝐧𝐞𝐝 a scalable classifier using an image retrieval model that can dynamically handle new classes without additional training.

➞ 𝐈𝐧𝐜𝐫𝐞𝐚𝐬𝐞𝐝 the 𝐫𝐞𝐩𝐫𝐨𝐝𝐮𝐜𝐢𝐛𝐢𝐥𝐢𝐭𝐲 of the 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐩𝐫𝐨𝐜𝐞𝐬𝐬 by implementing MLOps solutions, such as DVC & MLFlow, that included data & model versioning, registries, and experiment tracking features.

Continental

🟡 Autonomous driving system using computer vision & deep learning

The client, a well-known player in the automotive sector, needed to create an autonomous driving car prototype that could accurately detect traffic participants by leveraging DL and CV solutions.

➞ 𝐋𝐞𝐝 the 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 of a prototype for real-time traffic participants detection by deploying a CenterNet model for 3D object detection on a test car using an Nvidia Drive AGX C++ platform.

➞ 𝐈𝐦𝐩𝐫𝐨𝐯𝐞𝐝 the reproducibility of the experimentation process by implementing MLOps solutions (DVC) that version unstructured big data.

➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a replacement to the Hungarian algorithm that 𝐢𝐧𝐜𝐫𝐞𝐚𝐬𝐞𝐝 the 𝐬𝐩𝐞𝐞𝐝 by 𝟑𝟎%, making the system 𝐫𝐞𝐚𝐥-𝐭𝐢𝐦𝐞.

➞ 𝐒𝐚𝐯𝐞𝐝 𝟐 𝐝𝐚𝐲𝐬, in 𝐞𝐯𝐞𝐫𝐲 𝐦𝐨𝐧𝐭𝐡, of 𝐫𝐞𝐩𝐞𝐭𝐢𝐭𝐢𝐯𝐞 work by automating the steps for the creation of the 𝐦𝐨𝐧𝐭𝐡𝐥𝐲 demo.

Mid-Level Machine Learning Researcher

⚫️ Autonomous driving research using computer vision & deep learning

The client, a well-known player in the automotive sector, needed to expand its research into the autonomous driving field using novel deep learning and computer vision methods.

➞ 𝐑𝐞𝐬𝐞𝐚𝐫𝐜𝐡 in 3D object detection, depth estimation & completion, tracking, and sensor fusion algorithms using various sensors such as images, radar & LiDAR point clouds.

➞ 𝐑𝐞𝐝𝐮𝐜𝐞𝐝 𝐛𝐲 𝟒𝟎% the 𝐦𝐞𝐦𝐨𝐫𝐲 𝐮𝐬𝐚𝐠𝐞 of the 3D object detection code by introducing the Dynamic Voxelization algorithm into the preprocessing pipeline.

➞ 𝐓𝐫𝐚𝐢𝐧𝐞𝐝 state-of-the-art models to generate LiDAR-like dense point clouds from sparse Radar point clouds on various autonomous driving datasets, such as KITTI, nuScenes, DDAD, and A2D2.

Junior Machine Learning Researcher

Elementum Technologies

⚫️ Dorel - An Uber for handymen

𝐃𝐨𝐫𝐞𝐥 is an app that connects handypersons with customers. It contains a review and portfolio system where any handyperson can build his profile. Based on that, a user can determine whether he fits the job. The app intends to bring, once again, trust in handypersons.

➞ 𝐏𝐢𝐭𝐜𝐡𝐞𝐝 the 𝐢𝐝𝐞𝐚 to various start-up incubators.

➞ 𝐌𝐚𝐧𝐚𝐠𝐞𝐝 the backend components of the software solution of Dorel using an Agile methodology.

➞ 𝐋𝐞𝐝 the 𝐝𝐞𝐬𝐢𝐠𝐧 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 of the Django RESTful API server and the notifications microservice.

➞ 𝐃𝐞𝐬𝐢𝐠𝐧𝐞𝐝 and 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐝 the Redux layer of the React Native mobile app.

Co-Founder

SafeFleet Telematics

⚫️ Smart parking management platform

The client, a local player in the urban parking payment systems, needed to scale its operations into the private sector by building a new parking platform. 

➞ 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐞𝐝 the Redux layer of the onboarding mechanism of the React Native mobile application.

➞ 𝐁𝐮𝐢𝐥𝐭 an autonomous system that could detect license numbers and automatically open the parking barrier for registered users using MQTT.

➞ 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐝 the Python async notifications microservice for the Android & IOS mobile apps.

Mid-Level Software Developer

⚫️ Smart parking management platform

The client, a local player in the urban parking payment systems, needed to scale its operations into the private sector by building a new parking platform. 

➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a suite of unit tests that 𝐝𝐞𝐜𝐫𝐞𝐚𝐬𝐞𝐝 the 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 𝐭𝐢𝐦𝐞 by 𝟐𝟎%.

➞ 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐝 various Django RESTful API server features, such as new endpoints, serializers, and CRUD operations.

➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a real-time business metrics dashboard on the office TV  using Grafana on a Raspberry Pi.

Junior Backend Developer

🤖 Senior Machine Learning Engineer • MLOps • Contractor ~ Helping startups engineer production-ready ML/AI systems.

I am a senior machine learning engineer and contractor with 𝟲+ 𝘆𝗲𝗮𝗿𝘀 𝗼𝗳 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲. I design and implement modular, scalable, and production-ready ML systems for startups worldwide.

My central mission is to build data-intensive AI/ML products that serve the world.

Since training my first neural network in 2017,  I have 2 passions that fuel my mission:

→ Designing and implementing production AI/ML systems using MLOps best practices.

→ Teaching people about the process.

.

I currently develop production-ready Deep Learning products at Metaphysic, a leading GenAI platform. In the past, I built Computer Vision and MLOps solutions for CoreAI, Everseen, and Continental.

Also, I am the Founder of Decoding ML, a channel for battle-tested content on learning how to design, code, and deploy production-grade ML and MLOps systems.

I am writing articles and posts each week on:

- 𝘓𝘪𝘯𝘬𝘦𝘥𝘐𝘯: 29k+ followers
- 𝘔𝘦𝘥𝘪𝘶𝘮: 2.5k+ followers ~ 🔗 https://medium.com/@pauliusztin
- 𝘚𝘶𝘣𝘴𝘵𝘢𝘤𝘬 (𝘯𝘦𝘸𝘴𝘭𝘦𝘵𝘵𝘦𝘳): 6k+ followers ~ 🔗 https://decodingml.substack.com/

.

If you want to learn how to build an end-to-end production-ready LLM & RAG system using MLOps best practices, you can take Decoding ML’s self-guided free course:

→ 𝘓𝘓𝘔 𝘛𝘸𝘪𝘯 𝘊𝘰𝘶𝘳𝘴𝘦: 𝘉𝘶𝘪𝘭𝘥𝘪𝘯𝘨 𝘠𝘰𝘶𝘳 𝘗𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯-𝘙𝘦𝘢𝘥𝘺 𝘈𝘐 𝘙𝘦𝘱𝘭𝘪𝘤𝘢 ~ 🔗 https://github.com/decodingml/llm-twin-course

.

💬 If you need machine learning solutions for your business, let’s discuss!

🌎 Only open to full remote positions as a contractor.

.

Contact:

📱 Phone: +40 732 509 516

✉️ Email: p.b.iusztin@gmail.com

💻 Decoding ML: https://linktr.ee/decodingml

🕵🏻‍♂️ Personal site & Socials: https://www.pauliusztin.me/

🟡 Scalable ML batch infrastructure for continuous training & inference

The client, a generative AI leader, struggled with manual training and calling their ML models. As a tedious and unscalable process, they needed to automate their ML processes completely and integrate them with their current web platform.

➞ 𝐃𝐞𝐬𝐢𝐠𝐧𝐞𝐝 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a modular 𝐏𝐲𝐭𝐡𝐨𝐧 𝐩𝐚𝐜𝐤𝐚𝐠𝐞 that composes the ML model required steps into 3 main batch pipelines that can easily be automated: feature, training, and inference, 𝐞𝐥𝐢𝐦𝐢𝐧𝐚𝐭𝐢𝐧𝐠 a 𝐭𝐞𝐝𝐢𝐨𝐮𝐬 𝟐-𝐡𝐨𝐮𝐫 𝐦𝐚𝐧𝐮𝐚𝐥 𝐩𝐫𝐨𝐜𝐞𝐬𝐬/𝐯𝐢𝐝𝐞𝐨.
➞ 𝐃𝐞𝐩𝐥𝐨𝐲𝐞𝐝 a scalable & cost-effective 𝐛𝐚𝐭𝐜𝐡 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 and 𝐂𝐈/𝐂𝐃 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 on top of AWS ECS & SQS that scales in and out based on the number of jobs, 𝐬𝐚𝐯𝐢𝐧𝐠 𝐭𝐡𝐨𝐮𝐬𝐚𝐧𝐝𝐬 𝐨𝐟 𝐝𝐨𝐥𝐥𝐚𝐫𝐬/𝐦𝐨𝐧𝐭𝐡.
➞ 𝐒𝐚𝐯𝐞𝐝 𝐦𝐨𝐧𝐭𝐡𝐬 𝐨𝐟 𝐟𝐮𝐭𝐮𝐫𝐞 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 by using good software practices to make the Python module & batch cloud infrastructure flexible and composable.
➞ 𝐁𝐮𝐢𝐥𝐭 a 𝐯𝐨𝐢𝐜𝐞 𝐬𝐲𝐧𝐭𝐡𝐞𝐬𝐢𝐬 𝐏𝐨𝐂 in ~1 week required to 𝐜𝐥𝐨𝐬𝐞 the 𝐝𝐞𝐚𝐥 with one of the 𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐜𝐥𝐢𝐞𝐧𝐭𝐬.

⚫️ Big data ingestion & archiving pipeline

The client wanted to expand its platform features by securely and safely ingesting, preprocessing, and storing TB of data into their platform.

➞ 𝐂𝐨𝐧𝐭𝐫𝐢𝐛𝐮𝐭𝐞𝐝 to the 𝐝𝐞𝐬𝐢𝐠𝐧 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 of a 𝐝𝐚𝐭𝐚 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐧𝐠 >𝟓 𝐓𝐁 of data in 𝐨𝐧𝐞 𝐠𝐨 in a reliable, fast, and secure way.
➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a scalable 𝐝𝐚𝐭𝐚 𝐩𝐫𝐞𝐩𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports multiple media types (images, videos, audio, 3D objects) and creates small preview files that are cheap to transfer, store and visualize. The 𝐨𝐫𝐢𝐠𝐢𝐧𝐚𝐥 𝐟𝐢𝐥𝐞𝐬 are 𝐚𝐫𝐜𝐡𝐢𝐯𝐞𝐝 on an affordable storage solution 𝐫𝐞𝐝𝐮𝐜𝐢𝐧𝐠 𝐜𝐨𝐬𝐭𝐬 𝐛𝐲 𝐮𝐩 𝐭𝐨 𝟖𝟏%.

If you are into LLMs and feel lost (like most of us), here is a fantastic repository that curates the top tools, blogs and tutorials.

The LLM space is complicated and moves fast...

Thus, it can often make us feel lost and confused.

That's why Li Yin started an excellent initiative by creating the LLM-engineer-handbook repository, which curates and aggregates the top:

- tools
- blogs
- tutorials
- books
- social accounts

On LLM topics such as:

- training
- fine-tuning
- serving
- building LLM apps

Consider checking out her repository and support it with a 🌟

💻 Repo: https://lnkd.in/d4p7ZAMJ

Let's make this the go-to place for finding resources for building LLM apps!

P.S. The name has nothing to do with my book but is on her list.

#machinelearning #artificialintelligence #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

Senior ML/AI Engineer • MLOps • Founder @ Decoding ML ~ Posts and articles about building production-grade ML/AI systems.

Here is one of the best books to start in #AI engineering and #GenAI

I've recently read parts of the "Building LLMs for Production" book by Louis-François Bouchard and Louie Peters and loved it!

I think it's the perfect book for people getting into GenAI, LLMs and RAG, as it lays down the foundations (in a very practical way) for aspects such as:

- Reducing hallucinations for LLMs
- Prompt engineering
- Advanced RAG (chunking, optimization, evaluation)
- Agents
- Fine-tuning
- LLM evaluation

Enough to get a strong sense of how GenAI and LLMs work in an industry setup.

Most chapters are backed up by small projects/tutorials built in LlamaIndex (or LangChain), DeepLake and OpenAI.

Here are a few tutorials that I am excited about:

- Creating Knowledge Graphs from textual data
- Multimodal financial document analysis from PDFs
- Improving LLMs with RLHF

While reading the book, you will understand how to use LlamaIndex and LangChain to build AI solutions.

Kudos to Louis and Louie for this masterpiece!

.

I think it works perfectly with our LLM Engineer's Handbook. Here is how I would approach getting into the field:

1. Start with `Building LLMs for Production` to get a strong sense of the field and what problems it can solve

🔗 https://lnkd.in/djxAPQQA

2. Wrap up with our `LLM Engineer's Handbook` that will teach you an end-to-end framework for building LLM apps using LLMOps best practices.

🔗 https://lnkd.in/dVgFJtzF

...and boom, you are in the game of AI engineering.

#machinelearning #artificialintelligence #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

No alternative text description for this image

A blueprint for designing production LLM systems: From Notebooks to production 
 
For example, we will fine-tune an LLM and do RAG on social media data, but it can easily be adapted to any data. 
 
We have 4 core components. 
 
We will follow the feature/training/inference (FTI) pipeline architecture. 
 
𝟭. 𝗗𝗮𝘁𝗮 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 
 
It is based on an ETL that: 
- crawls your data from blogs and socials 
- standardizes it 
- loads it to a NoSQL database (e.g., MongoDB) 
 
As: 
- we work with text data, which is naturally unstructured 
- no analytics required 
 
→ a NoSQL database fits like a glove.

 
𝟮. 𝗙𝗲𝗮𝘁𝘂𝗿𝗲 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 
 
It takes raw articles, posts and code data points from the data warehouse, processes them, and loads them into a logical feature store. 
 
Let's focus on the logical feature store. As with any RAG-based system, a vector database is one of the central pieces of the infrastructure. 
 
We directly use a vector database as a logical feature store. 
 
Unfortunately, the vector database doesn't offer the concept of a training dataset. 
 
To implement this, we will wrap the retrieved data into a versioned, tracked, and shareable MLOps artifact. 
 
To conclude: 
 
- the training pipeline will use the instruct datasets as artifacts (offline) 
- the inference pipeline will query the vector DB for RAG (online) 
 
 
𝟯. 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 
 
It consumes instruct datasets from the feature store, fine-tunes an LLM with it, and stores the tuned LLM weights in a model registry. 
 
More concretely, when a new instruct dataset is available in the logical feature store, we will trigger the training pipeline, consume the artifact, and fine-tune the LLM. 
 
We run multiple experiments to find the best model and hyperparameters. We will use an experiment tracker to compare and select the best hyperparameters. 
 
After the experimentation phase, we store and reuse the best hyperparameters for continuous training (CT). 
 
The LLM candidate's testing pipeline is triggered for a detailed analysis. If it passes, the model is tagged as accepted and deployed to production. 
 
Our modular design lets us leverage an ML orchestrator to schedule and trigger the pipelines for CT. 
 
 
𝟰. 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 
 
It is connected to the model registry and logical feature store. From the model registry, it loads a fine-tuned LLM, and from the logical feature store, it accesses the vector DB for RAG. 
 
It receives client requests as queries through a REST API. It uses the fine-tuned LLM and vector DB to do RAG to answer the queries. 
 
Everything is sent to a prompt monitoring system to analyze, debug, and understand the system. 

#artificialintelligence #machinelearning #mlops

Here is a Python code base template that will help you follow best MLOps practices since the beginning of your project ↓

I recently discovered a GitHub repository that is a fantastic source to learn and inspire on how to:

- structure your Python project
- implement MLOps best practices
- integrate various MLOps tools

.

More concretely, it gives you the boiler code for:

- CI/CD
- QA (Ruff, Bandit, PyTest, MyPy, testing coverage)
- Config system in YAML format (OmegaConf, PyDantic)
- Data validation (PyDantic, Pandera)
- Python packaging and environment management (Poetry, Docker)
- model registries, experiment tracking (MLFlow)
- unit & integration tests for your ML pipelines
- hyperparameter optimization (Optuna)

...and the nice part is that everything is structured using SWE best practices, such as:

- DAGs (aka pipelines)
- SOLID principles
- IO Separation
- Soft Condig

.

Here is the repo developed by Médéric HURIER

🔗 𝘔𝘓𝘖𝘱𝘴 𝘗𝘺𝘵𝘩𝘰𝘯 𝘗𝘢𝘤𝘬𝘢𝘨𝘦: https://lnkd.in/dhA-E84u

It's worth looking at (even for more experienced MLE / MLOps people)!

#artificialintelligence #machinelearning #mlops

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

graphical user interface, text, application, chat or text message

I will host a live workshop based on my LLM Engineer's Handbook, where I will go over the architecture of the LLM Twin system presented throughout the book.

The goal of the presentation is to connect the dots between:

- DE
- ML research/DS
- MLE/AIE
- MLOps/LLMOps

By the end of the workshop, you will have a strong intuition of what it takes to implement an end-to-end LLM system, from data collection to production.

The live workshop will be virtual and FREE.

Hosted this Thursday (Nov 21) at 7 PM CET (12 PM CST) by the The Data Entrepreneurs

🔗 Event link: https://lnkd.in/dr45zUf9

You can also join a pre-event (6:30 OM CET) and post-event (8:07 PM CET) huddle on Discord at https://lnkd.in/d35C5fYN

Thank you, Shaw Talebi, for hosting this webinar and allowing me to present an overview of my LLM Engineer's Handbook!

#artificialintelligence #machinelearning #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

Build your real-time personalized recommender from scratch! 
We just released a free course on building an H&M real-time personalized recommender.

The 5-lesson course has written tutorials, notebooks, Python code, and cloud deployments.

.

> What you'll learn

The course will focus on engineering a production-ready recommender system touching:

- ML system design
- the architecture of a real-time personalized recommender
- MLOps best practices (feature store, model registry, serving)

> What you'll build

- A production-ready recommender system with real-time personalization
- 4-stage recommender architecture for instant recommendations
- Two-tower model for user and fashion item embeddings
- LLM-enhanced recommendation system

> Tech stack & deployment

- Feature engineering with Polars
- Real-time serving with KServe
- MLOps infrastructure using the Hopsworks AI Lakehouse
- Offline batch deployments using GitHub Actions
- Interactive UI with Streamlit Cloud
- dependency management using `uv`

The outcome will be an H&M deployed personalized recommender you can tinker with.  

.

I've learned a lot from my past open-source courses.

Together with Hopsworks, we've made something special.

💻 Access the code and lessons: https://lnkd.in/dqs8bzXA

#machinelearning #artificialintelligence #mlops

Fantastic news! I remastered the LLM Twin open-source course to make everything easier to run, added AWS SageMaker for training and inference, and Opik for LLM evaluation and prompt monitoring.

I got a lot of feedback from people doing the course that it is:

- hard to set everything up
- the structure is unclear
- there is too much noise in the files (duplicates, etc.)
- bugs in the code

I took that advice seriously and refactored the whole course. Now:

- we have a single Makefile
- you can spin up the whole infrastructure with `make local-start`
- the folder structure follows the course lessons
- everything runs smoothly (at least on macOS and Linus machines)

.

𝗤𝘂𝗶𝗰𝗸 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗟𝗟𝗠 𝗧𝘄𝗶𝗻 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝗰𝗼𝘂𝗿𝘀𝗲 ↓

It consists of:

- 12 written lessons
- open-source code to run everything yourself 

→ 𝘊𝘰𝘮𝘱𝘭𝘦𝘵𝘦𝘭𝘺 𝘧𝘳𝘦𝘦, 𝘯𝘰 𝘩𝘪𝘥𝘥𝘦𝘯 𝘤𝘰𝘴𝘵𝘴.

𝗕𝘆 𝗳𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 the 𝗟𝗟𝗠 𝗧𝘄𝗶𝗻 𝗳𝗿𝗲𝗲 𝗰𝗼𝘂𝗿𝘀𝗲, you will learn how to design, train, and deploy a production-ready LLM Twin of yourself powered by LLMs, vector DBs, RAG and LLMOps good practices.

You will 𝗹𝗲𝗮𝗿𝗻 how to architect and build a real-world LLM & RAG system from start to finish - from data collection to deployment.

You will also 𝗹𝗲𝗮𝗿𝗻 to leverage MLOps best practices, such as experiment trackers, model registries, prompt monitoring, and versioning.

𝗪𝗵𝘆 𝘀𝗵𝗼𝘂𝗹𝗱 𝘆𝗼𝘂 𝗰𝗮𝗿𝗲? 🫵

→ No more isolated scripts or Notebooks! Learn production ML by building and deploying an end-to-end production-grade LLM system.

.

The architecture of the LLM twin is split into 4 Python pipelines:

- data collection
- feature engineering
- training
- real-time inference

Along the 4 microservices, you will learn to integrate 4 AI tools:

- @company_comet-ml as your ML platform
- 𝗢𝗽𝗶𝗸 as your prompt evaluation and monitoring tool
- @company_qdrant as your vector DB
- 𝗔𝗪𝗦 𝗦𝗮𝗴𝗲𝗠𝗮𝗸𝗲𝗿 as your ML infrastructure

.

Start learning by checking out the repository:

🧑‍💻 https://lnkd.in/dzat6PB6

...and the free lessons:

📙 https://lnkd.in/dX__4mhX

#artificialintelligence #machinelearning #generativeai #mlops

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

This is how I stay sane in the AI and LLM world.

As much as I love coding, I still need a day a week to go outside and smell the flowers. 

#artificialintelligence #machinelearning #programming

Evaluating LLMs & RAG is hard. You must pick the right metrics, track chains and compare experiments. Here, one tool that will save you ↓

Here is 𝗼𝗻𝗲 𝘁𝗼𝗼𝗹 𝘁𝗼 𝗿𝘂𝗹𝗲 𝘁𝗵𝗲𝗺 𝗮𝗹𝗹: Opik ←

An 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 #llm 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗽𝗿𝗼𝗺𝗽𝘁 𝗺𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 𝘁𝗼𝗼𝗹 powered by Comet

𝘖𝘱𝘪𝘬 𝘪𝘴 𝘓𝘓𝘔 𝘢𝘯𝘥 𝘓𝘓𝘔𝘖𝘱𝘴-𝘤𝘦𝘯𝘵𝘳𝘪𝘤. 𝘛𝘩𝘶𝘴...

1/ From a minimalist UI, you can track your evaluation datasets, create multiple experiments, log your metrics, and investigate your prompt chains.

2/ They provide out-of-the-box heuristic and LLM-judge metrics that you can leverage in your app and track in their dashboard: 🔗 https://lnkd.in/e8H-Ficj

3/ You can manage and version your prompts from their prompt registry.

4/ You can easily monitor your prompt chains by decorating your Python functions with @𝘰𝘱𝘪𝘬.𝘵𝘳𝘢𝘤𝘬().

5/ You can use it to gather human feedback scores for preference alignment fine-tuning.

6/ The cherry on top is that you can easily integrate it with LLM tools, such as OpenAI, Antrophic, Groq and more!

7/ You can integrate it with Ragas for RAG evaluation.

I've used Opik in my LLM Engineer's Handbook and LLM Twin course and loved their minimalistic UI, intuitive Python SDK and the features enumerated above,

🔗 𝘍𝘪𝘯𝘥 𝘮𝘰𝘳𝘦 𝘢𝘣𝘰𝘶𝘵 𝘖𝘱𝘪𝘬: https://lnkd.in/eFhbz5e2

#machinelearning #artificialintelligence #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

My workshop on my LLM Engineer's Handbook was a blast. It was my first webinar streamed to LI and YT, having 150 > viewers. 

🔗 You can find the recorded presentation on YouTube at: https://lnkd.in/dayS7gh4

.

The goal of the presentation is to connect the dots between:

- DE
- ML research/DS
- MLE/AIE
- MLOps/LLMOps

By the end of the workshop, you will have a strong intuition of what it takes to implement an end-to-end LLM system, from data collection to production.

.

Thank you, Shaw Talebi, for inviting and hosting this fantastic event! You have opened my mind to doing more live events!

#machinelearning #artificialintelligence #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

The difference between 𝗲𝗻𝗰𝗼𝗱𝗲𝗿𝘀, 𝗱𝗲𝗰𝗼𝗱𝗲𝗿𝘀 and 𝗲𝗻𝗰𝗼𝗱𝗲𝗿-𝗱𝗲𝗰𝗼𝗱𝗲𝗿 𝗟𝗟𝗠𝘀. 
 
Embeddings are everywhere... both encoders and decoders use self-attention layers to encode word tokens into embeddings. 
 
Let's see when to use each architecture ↓ 
 
The key difference between an encoder & decoder is in how it processes its inputs & outputs. 
 
=== 𝗘𝗻𝗰𝗼𝗱𝗲𝗿𝘀 === 
 
The role of an encoder is to extract relevant information from the whole input and encode it into an embedding (e.g., BERT, RoBERTa). 
 
Within the "Multi-head attention" of the transformer, all the tokens are allowed to speak to each other. 
 
A token at position t can talk to all other previous tokens [0, t-1] and future tokens [t+1, T]. This means that the attention mask is computed along the whole vector. 
 
Thus, because the encoder processes the whole input, it is helpful for classification tasks (e.g., sentiment analysis) and creates embeddings for clustering, recommender systems, vector DB indexes, etc. 
 
=== 𝗗𝗲𝗰𝗼𝗱𝗲𝗿𝘀 === 
 
On the flip side, if you want to generate text, use decoder-only models (e.g., GPT family). 
 
Only the current and previous tokens (not the whole input) are used to predict the next token. 
 
Within the "Masked Multi-head attention," the future positions are masked to maintain the autoregressive property of the decoding process. 
 
For example, within the "Masked Multi-head attention," instead of all the tokens talking to each other, a token at position t will have access only to previous tokens at positions t-1, t-2, t-3, ..., 0. 
 
=== 𝗘𝗻𝗰𝗼𝗱𝗲𝗿-𝗱𝗲𝗰𝗼𝗱𝗲𝗿 === 
 
This technique is used when you have to understand the entire input sequence (encoder) and the previously generated sequence (decoder -> autoregressive). 
 
Typical use cases are text translation & summarization (the original transformer was built for text translation), where the output heavily relies on the input. 
 
Why? Because the decoding step always has to be conditioned by the encoded information. Also known as cross-attention, the decoder queries the encoded information for information to guide the decoding process. 
 
For example, when translating English to Spanish, every Spanish token predicted is conditioned by the previously predicted Spanish tokens & the entire English sentence. 
 
. 
 
To conclude... 
 
- a decoder takes as input previous tokens and predicts the next one (in an autoregressive way) 
 
- by dropping the "Masked" logic from the "Masked Multi-head attention," you process the whole input, transforming the decoder into an encoder 
 
- if you hook the encoder to the decoder through a cross-attention layer, you have an encoder-decoder architecture 

#machinelearning #artificialintelligence #generativeai

The engineer's guide to LLM & RAG evaluation

Here's the hard truth: If you can't measure your LLM's efficiency, you can't improve it.

Before diving into optimizations, street-smart engineers focus on two things:
1. Building a rapid end-to-end prototype
2. Creating a robust evaluation pipeline

Before starting building, always ask yourself what metrics you need to measure...

...as that will guide you in maximizing it.

Start with metrics that matter to YOUR use case:
- Cost efficiency
- Output quality
- Classification accuracy
- Response time
- ... or whatever drives value for your users

Remember: These metrics will drive every decision you make.

So pick carefully!

.

Now, let's break down the 4 critical components of LLM evaluation:

- input
- predictions / generated output
- ground truth (GT)
- RAG context

.

Based on what you have at your disposal, here is how you can combine them: 

1/ 𝗛𝗲𝘂𝗿𝗶𝘀𝘁𝗶𝗰𝘀 (Levenshtein, perplexity, BLEU and ROUGE) and similarity scores (BERT Score) between the 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝗼𝗻𝘀 and 𝗚𝗧 (similar to classic metrics).

2/ 𝗟𝗟𝗠-𝗮𝘀-𝗷𝘂𝗱𝗴𝗲𝘀 to evaluate for hallucination and moderation, based solely on the user's 𝗶𝗻𝗽𝘂𝘁 and 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝗼𝗻𝘀.

3/ 𝗟𝗟𝗠-𝗮𝘀-𝗷𝘂𝗱𝗴𝗲𝘀 to evaluate for hallucination and moderation based on the user's 𝗶𝗻𝗽𝘂𝘁, 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝗼𝗻𝘀 and 𝗚𝗧.

4/ 𝗟𝗟𝗠-𝗮𝘀-𝗷𝘂𝗱𝗴𝗲𝘀 to evaluate the RAG pipeline on problems such as recall and precision based on the user's 𝗶𝗻𝗽𝘂𝘁, 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝗼𝗻𝘀, 𝗚𝗧, and the 𝗥𝗔𝗚 𝗰𝗼𝗻𝘁𝗲𝘅𝘁.

5/ Implementing 𝗰𝘂𝘀𝘁𝗼𝗺 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that leverage points 1 to 4. 

.

Heuristic metrics don't work well when assessing GenAI systems as they measure exact matches between the generated output and GT. 

...they don't consider synonyms or that two sentences share the same idea but use entirely different words.

So...

LLM systems are primarily evaluated with similarity scores and LLM judges.

1/ LLM judges are great as they are super versatile, and in most cases, you don't need a GT to compute the metrics.

2/ But they are very costly. Thus, adopting similarity scores is still a valid option.

.

To see an end-to-end implementation of evaluating LLM & RAG applications using Opik (by Comet) on:

- Hallucination
- Moderation
- Custom business metrics
- Context recall and precision

📙 Read this article from the LLM Twin course: https://lnkd.in/d4PVQ6hc

#machinelearning #artificialintelligence #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

The whole field of prompt engineering can be reduced to these 6 techniques I use almost daily when using ChatGPT (or other LLMs). 
 
#1. Few shot prompting 
 
Add 2 or 3 high-quality demonstrations of the target task to your prompt, each consisting of input and desired output. 
 
The LLM will better understand your intention and what kind of answers you expect based on concrete examples. 
 
#2. Self-consistency sampling 
 
Sample multiple outputs with "temperature > 0" and select the best one out of these candidates. 
 
How to pick the best candidate? 
 
It will vary from task to task, but here are 2 primary scenarios ↓ 
 
1. Some tasks are easy to validate, such as programming questions. In this case, you can write unit tests to verify the correctness of the generated code. 
 
2. For more complicated tasks, you can manually inspect them or use another LLM (or another specialized model) to rank them. 
 
#3. Chain-of-Thought (CoT) 
 
You want to force the LLM to explain its thought process, which eventually leads to the final answer, step by step. 
 
This will help the LLM to reason complex tasks better. 
 
You want to use CoT for complicated reasoning tasks + large models (e.g., with more than 50B parameters). Simple tasks only benefit slightly from CoT prompting. 
 
Here are a few methods to achieve CoT: 
- provide a list of bullet points with all the steps you expect the LLM to take 
- use "Few shot prompt" to teach the LLM to think in steps 
 
... or my favorite: use sentences such as "Let's think step by step." 
 
#4. Augmented Prompts 
 
The LLM's internal knowledge is limited to the data on which it was trained. It often forgets specific details of older training datasets. 
 
The most common use case is Retrieval-Augmented Generation (RAG). 
 
That is why using the LLM as a reasoning engine is beneficial. It allows one to parse and extract information from a reliable source given as context in the prompt. 
 
𝘞𝘩𝘺? 
- avoid retraining the model on new data 
- avoid hallucinating 
- access to references on the source 
 
#5. A single responsibility per prompt 
 
Quite self-explanatory. It is similar to the DRY principle in SWE. 
 
Having only x1 task/prompt is good practice to avoid confusing the LLM. 
 
If you have more complex tasks, split them into granular ones and merge the results later in a different prompt. 
 
#6. Be as explicit as possible 
 
The LLM cannot read your mind. To maximize the probability of getting precisely what you want, you can imagine the LLM as a 7-year-old to whom you must explain everything step-by-step.
 
. 
 
The truth is that prompt engineering is quite intuitive. We don't have to overthink it too much. 
 
What would you add to this list? 

#artificialintelligence #machinelearning #generativeai

RAG is everywhere and a critical component in most gen AI applications. But how do you evaluate your RAG pipeline?

This framework provides a clear mindmap for evaluating your RAG application.

→ RAGAs

Let's see what the RAGAs framework offers ↓

A RAG application has 3 major components:

1. 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗗𝗕: holds all the context your application needs to generate valuable answers (usually implemented as a vector DB)

2. 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗲𝗿: takes care of querying the Knowledge DB and retrieves additional context that matches the user's query.

3. 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗼𝗿: takes in the user's input and the retrieved context. It uses an LLM to generate the final answer

You must evaluate both:
- the retrieved context
- the generated answer

.

RAGAs propose the following RAG evaluation framework

↓↓↓

You have to check the accuracy of the following 3 elements (individually and the interaction between each other):

- user input
- context (retrieved based on the user's query)
- generated answer

Intuitively, you want to see that:

- the context is relevant to the user's input and contains all the necessary information
- the generated answer is relevant to the context and user input and doesn't contain any hallucinations

.

These are the official metrics that RAGAs provide:

1. For the 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 :

𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗣𝗿𝗲𝗰𝗶𝘀𝗶𝗼𝗻
Evaluates the precision of the context used to generate an answer, ensuring relevant information is selected from the context.

𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗥𝗲𝗰𝗮𝗹𝗹
Measures if all the relevant information required to answer the question was retrieved.

𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝘁𝗶𝘁𝗶𝗲𝘀 𝗥𝗲𝗰𝗮𝗹𝗹
Evaluates the recall of entities within the context, ensuring that no important entities are overlooked.

2. For the 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻:

𝗙𝗮𝗶𝘁𝗵𝗳𝘂𝗹𝗻𝗲𝘀𝘀
Measures how accurately the generated answer reflects the source content, ensuring the generated content is truthful and reliable.

𝗔𝗻𝘀𝘄𝗲𝗿 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲
It is validating that the response directly addresses the user's query.

𝗔𝗻𝘀𝘄𝗲𝗿 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗦𝗶𝗺𝗶𝗹𝗮𝗿𝗶𝘁𝘆
Shows that the generated content is semantically aligned with expected responses.

𝗔𝗻𝘀𝘄𝗲𝗿 𝗖𝗼𝗿𝗿𝗲𝗰𝘁𝗻𝗲𝘀𝘀
Focuses on fact-checking, assessing the factual accuracy of the generated answer.

#machinelearning #artificialintelligence #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

It's so exciting to see that the LLM Engineer's Handbook is still a best seller in multiple categories on Amazon, such as:

- Natural Language Processing (Books): https://lnkd.in/dt-NbSBa
- Computer Neural Networks:  https://lnkd.in/dE5epaMf

I am super proud to see it in the top 10 in the AI section, competing with masterpieces such as Nexus by Harari:  https://lnkd.in/dTcXJppm

I am not the type of guy who brags, but I am excited about how well the LLM community has received the book.

Initially, I was a little nervous as we took this different approach, showing the whole book how to build a single project (but end-to-end, teaching people what an industry-level project looks like).

Usually, books (with or without code) focus on small projects or more granular concepts. 

But seeing that people liked this approach was fantastic and a solid signal to continue my current strategy.

→ As most of my current and future content will take the same approach.

Building cool, industry-level #llm & #recsys apps!

📙 Amazon: https://lnkd.in/dVgFJtzF
💻 GitHub: https://lnkd.in/dcp9yEyX

#artificialintelligence #machinelearning #generativeai

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

Unlock 𝘀𝘂𝗽𝗲𝗿𝗶𝗼𝗿 𝘁𝗲𝘅𝘁 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 for 𝗟𝗟𝗠𝘀 by 𝗺𝗮𝘀𝘁𝗲𝗿𝗶𝗻𝗴 these 𝟰 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗱𝗲𝗰𝗼𝗱𝗶𝗻𝗴 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀. 
 
You see, LLMs don't just spit out text. 
 
They calculate "logits", which are mapped to probabilities for every possible token in their vocabulary. 
 
It uses previous token IDs to predict the next most likely token (the auto-regressive nature of decoder models). 
 
The real magic happens in the decoding strategy you pick ↓ 
 
1/ 𝗚𝗿𝗲𝗲𝗱𝘆 𝗦𝗲𝗮𝗿𝗰𝗵 
 
It only holds onto the most likely token at each stage. It's fast and efficient, but it is short-sighted. 
 
2/ 𝗕𝗲𝗮𝗺 𝗦𝗲𝗮𝗿𝗰𝗵 
 
This time, you are not looking at just the token with the highest probability. But you are considering the N most likely tokens. 
 
This will create a tree-like structure with N children for each node. 
 
The procedure repeats until you hit a maximum length or an end-of-sequence token. 
 
Ultimately, you pick the leaf with the biggest score and recursively pick its parent until you hit the root node. 
 
For example, in the graph below, we have "𝘣𝘦𝘢𝘮𝘴 = 2" and "𝘭𝘦𝘯𝘨𝘵𝘩 = 3". 
 
3/ 𝗧𝗼𝗽-𝗞 𝗦𝗮𝗺𝗽𝗹𝗶𝗻𝗴 
 
This technique extends the Beam search strategy and adds a dash of randomness to the generation process. 
 
Instead of just picking the most likely tokens, it's selecting a token randomly from the top k most likely choices. 
 
Thus, the tokens with the highest probability will appear more often, but other tokens will be generated occasionally to add some randomness ("creativity"). 
 
4/ 𝗡𝘂𝗰𝗹𝗲𝘂𝘀 𝗦𝗮𝗺𝗽𝗹𝗶𝗻𝗴 
 
In this case, you're not just picking the top k most probable tokens here. You're choosing a cutoff value p and forming a "nucleus" of tokens. 
 
In other words, rather than selecting the top k most probable tokens, nucleus sampling chooses a cutoff value p such that the sum of the probabilities of the selected tokens exceeds p. 
 
Thus, at every step, you will have various possible tokens included in the "nucleus" from which you sample. This introduces even more diversity and creativity into your output. 
 
. 
 
𝗡𝗼𝘁𝗲: For 𝘵𝘰𝘱-𝘬 and 𝘯𝘶𝘤𝘭𝘦𝘶𝘴 𝘴𝘢𝘮𝘱𝘭𝘪𝘯𝘨, you can also use the "𝘵𝘦𝘮𝘱𝘦𝘳𝘢𝘵𝘦" hyperparameter to tweak the output probabilities. It is a parameter that ranges from 0 to 1. A low temperature will decrease the entropy (randomness), making the generation more stable. 
 
. 
 
To summarize... 
 
There are 2 main decoding strategies: 
- greedy 
- beam 
 
To add more variability and creativity to beam search: 
- top-k 
- nucleus 

🔗 Learn more in this fantastic article written by Maxime Labonne: https://lnkd.in/d3WNAAFm

#artificialintelligence #machinelearning #generativeai

90% + of ML teams overlook the most crucial LLMOps metric: prompt monitoring. Here are 3 aspects production-ready systems track:

1/ Latency

LLM answers are usually streamed token-by-token. Thus, monitoring the latency of the full answer is not enough.

You have to compute:

- Time to First Token (TTFT)
- Time between Tokens (TBT)
- Tokens per Second (TPS)
- Time per Output Token (TPOT) 
- Total latency
- Input and output tokens count

2/ Metrics

Compute metrics that validate your model's performance for each input prompt and generated output tuple. 

Depending on your use case, you can compute % rates for:

- accuracy
- toxicity
- hallucination
- other business-specific metrics that ideally don't require labels

When working with RAG systems, you can also compute metrics relative to the relevance and precision of the retrieved context.

3/ Traces

Another essential thing to consider when monitoring prompts is logging their full traces. 

Multiple intermediate steps might be from the user query to the final general answer.

Thus, logging the full trace reveals the entire process from when a user sends a query to when the final response is returned, including the system's actions, the documents retrieved, and the final prompt sent to the model.

Additionally, you can log the latency, tokens, and costs at each step, providing a more fine-grained view of all the steps.

.

𝗧𝗼 𝗺𝗼𝗻𝗶𝘁𝗼𝗿 these 𝟯 𝗮𝘀𝗽𝗲𝗰𝘁𝘀, you 𝗻𝗲𝗲𝗱 𝘀𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝘁𝗼𝗼𝗹𝘀, such as Opik, an open-source tool maintained by Comet (already used by huge companies such as Uber and Netflix)

Here is a 𝗵𝗮𝗻𝗱𝘀-𝗼𝗻 𝗮𝗿𝘁𝗶𝗰𝗹𝗲 showing you 𝗵𝗼𝘄 𝘁𝗼 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗮 𝗺𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 using Opik:

🔗 https://lnkd.in/d4icAtxY

Enjoy!

#machinelearning #artificialintelligence #generativeai #mlops

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

AI/ML engineering is a combination of SWE and AI. So, stop playing with Notebooks and level up your SWE skills with this fantastic platform.

The only way to learn SWE is by building applications that are of different complexities.

One of my favorite platforms for learning SWE follows this same approach: CodeCrafters.io (YC S22)

They provide an interactive environment where you can code from scratch (in multiple languages, such as Python, Rust, and Java), well-known and loved tools.

Such as:

- Redis
- Shell
- BitTorrent
- SQLite
- Kafka

I've done their Redis series and loved it. When I have more time, I will use it to level up my Rust skills. 

The sad part is that they are not free. You have to pay a subscription.

The good part is that until the end of the year, you can get 𝗮 𝟳-𝗱𝗮𝘆 𝗳𝗿𝗲𝗲 𝘁𝗿𝗶𝗮𝗹 by signing up 𝘂𝘀𝗶𝗻𝗴 𝗺𝘆 𝗮𝗳𝗳𝗶𝗹𝗶𝗮𝘁𝗲 𝗹𝗶𝗻𝗸: 
https://lnkd.in/dQQm3bPV

Which is more than enough to go through a series, such as building Kafka! 

If you are disciplined and motivated enough 😂

Happy learning!

𝗡𝗼𝘁𝗲: If you subscribe, you also get 𝟰𝟬% 𝗼𝗳𝗳 using my affiliate link, or you can fully 𝗿𝗲𝗶𝗺𝗯𝘂𝗿𝘀𝗲 𝗶𝘁 through your corporate L&D budget.

#machinelearning #artificialintelligence #generativeai #mlops

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

𝗔𝘀𝗽𝗶𝗿𝗶𝗻𝗴 𝗔𝗜 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿? The truth is you need to master 2 basic skills before touching any ML models.

The truth about being a top AI & MLOps engineer is not sexy. 

In reality, you need 2 skills that have nothing to do with AI:

1. 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴: Learning Python is a perfect start.

2. 𝗖𝗹𝗼𝘂𝗱 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Learning to deploy your apps to a cloud such as AWS is critical in bringing value to your project.

𝘠𝘰𝘶 𝘯𝘦𝘦𝘥 𝘵𝘩𝘦𝘴𝘦 2 𝘣𝘦𝘧𝘰𝘳𝘦 𝘴𝘦𝘳𝘷𝘪𝘯𝘨 𝘔𝘓 𝘮𝘰𝘥𝘦𝘭𝘴 𝘪𝘯 𝘱𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯.

That's where the 𝗖𝗹𝗼𝘂𝗱 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗣𝘆𝘁𝗵𝗼𝗻 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 live course made by 🎧 Eric Riddoch kicks in.

I have known Eric for almost a year, and I can certify that he is a brilliant cloud, DevOps and MLOps engineer.

.

𝘏𝘦𝘳𝘦 𝘪𝘴 𝘸𝘩𝘢𝘵 𝘺𝘰𝘶 𝘸𝘪𝘭𝘭 𝘭𝘦𝘢𝘳𝘯 𝘥𝘶𝘳𝘪𝘯𝘨 𝘵𝘩𝘦 𝘤𝘰𝘶𝘳𝘴𝘦:

- enterprise-level AWS account management
- the fundamentals of cloud engineering
- design cloud-native RESTful APIs that cost-effectively scale from 4 to 4 million requests/day
- write, test, locally mock, and deploy code using the AWS SDK and OpenAI
- advanced observability and monitoring: logs, metrics, traces, and alerts

The principles learned during the course can easily be extrapolated from AWS to other cloud platforms such as GCP and Azure.

.

🕰️ The next cohort starts on Jan 6.

...and it provides:

- weekly classes
- lifetime access to self-paced videos, labs and articles
- certification of completion

 I strongly recommend it to lay down the SWE/DevOps foundations for AI/ML/MLOps engineering.

.


To 𝗴𝗲𝘁 𝟭𝟬% 𝗼𝗳𝗳, use the following 𝗰𝗼𝗱𝗲 when registering: DECODINGML

Eric also offers a 𝘀𝗰𝗵𝗼𝗹𝗮𝗿𝘀𝗵𝗶𝗽 𝗽𝗿𝗼𝗴𝗿𝗮𝗺 that can 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁𝗹𝘆 𝗿𝗲𝗱𝘂𝗰𝗲 𝘁𝗵𝗲 𝗽𝗿𝗶𝗰𝗲.

→ 💻  𝘌𝘯𝘳𝘰𝘭𝘭 𝘩𝘦𝘳𝘦: https://lnkd.in/dC_dJfiy

If you consider buying the course, please use the DECODINGML code to support my work.

#machinelearning #artificialintelligence #mlops

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

Flink and Kafka Streams will be replaced by these 2 streaming engines, which 100% integrate into Python's ecosystem for streaming applications.

For some quick context, a streaming system consists of 2 core components:

- a data persisting layer (e.g., Kafka, Red Pandas)
- a streaming engine (e.g., Flink, Kafka Streams)

→ The streaming engine reads data from the persisting layer's topics, processes it, and writes it to another topic.

Until now, the most popular choice for the streaming engine was Flink.

But for all the ML folks out there, it has a HUGE problem. 

It is based on JVM and dependent on the programming languages around it (e.g., Java, Scala, Kotlin). 

In most use cases, the Data Engineering team takes the prototype from the Data Science team and rewrites it in Flink.

Unfortunately, this rewriting process usually results in the training-serving skew, as the slightest difference in how the data is processed between training and inference can completely break the ML model.

The solution? Use a streaming engine that supports Python, which can be directly integrated with your ML training code.

.

𝘏𝘦𝘳𝘦 𝘢𝘳𝘦 𝘵𝘩𝘦 2 𝘣𝘪𝘨 𝘱𝘭𝘢𝘺𝘦𝘳𝘴 𝘸𝘩𝘦𝘯 𝘪𝘵 𝘤𝘰𝘮𝘦𝘴 𝘵𝘰 𝘴𝘵𝘳𝘦𝘢𝘮𝘪𝘯𝘨 𝘦𝘯𝘨𝘪𝘯𝘦𝘴 𝘪𝘯 𝘗𝘺𝘵𝘩𝘰𝘯 

↓↓↓

𝗕𝘆𝘁𝗲𝘄𝗮𝘅

- Is is built in Rust for speed and resource optimization.
- It exposes a Python interface to leverage it in AI applications fully.
- Follows an Object Oriented Programming (OOP) approach being suited for enterprise applications
- It has built-in connectors for Kafka, SQS, and more!

→🔗 https://bytewax.io/


𝗤𝘂𝗶𝘅 𝗦𝘁𝗿𝗲𝗮𝗺𝘀

- It is built in Python for ease of use and debugging.
- Follows almost 100% of Pandas's API, making it perfect for data scientists who work all day long with Pandas.
- Scalable as it leverages Kafka and K8s to provide data partitioning, consumer groups, state management and replication
- Has built-in Kafka connectors

→🔗 https://quix.io/

.

𝘛𝘰 𝘤𝘰𝘯𝘤𝘭𝘶𝘥𝘦:

- both follow the "Python. No JVM." paradigm
- Bytewax suits MLE and SWE, which want to build more complex applications.
- Quix is perfect for DS, which wants to leverage the power of a streaming engine.

#artificialintelligence #machinelearning #mlops

.

💡 Follow me for daily content on production GenAI, RecSys and MLOps.

Taplio

Paul Iusztin's Linkedin Analytics

What is Paul talking about?

Who is engaging with Paul

's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?