Aurimas Griciūnas

Senior Solutions Architect

neptune.ai

Chief Product Officer

Head of Product

Driving MLOps industry forward by helping Neptune customers succeed in adopting MLOps best practices.

SwirlAI

Driving MLOps, Data Engineering and Machine Learning industry forward by helping professionals Level Up in these disciplines.

Founder & CEO

Transactie Monitoring Nederland (TMNL)

Participated in building of ML Platform capabilities. Mainly focused on AWS Sagemaker and supporting AWS Services.

Senior MLOps Engineer

HomeToGo

- Led a Team of Cloud Engineers.
- Led a Team of Data Platform Engineers.
- Led Cloud Native Transformation Initiative.
- Led infrastructure configuration movement to IaaC (Terraform).

Data Architect / Data Platform & Cloud Engineering Lead

- Led a Team of Data Engineers.
- Led a Team of Data Platform Engineers.
- Initiated and led a successful company wide Cloud Native Transformation.
- Migrated entire Data Platform Stack to Kubernetes. This included:

        - Airflow.
        - Superset.
        - Spark.
        - MLFlow.
        - Flink.
        - Airbyte.
        - DataHub.

- Introduced first Stream Processing applications that moved some of the Data Warehouse Processing stages upstream the pipeline.

Senior Data Engineer / Data Platform Team Lead

- Led a team of Data Engineers.
- Initiated and led a Data Warehouse migration from Redshift to Snowflake effort. It also included a refactoring of the Data Model.
- Oversaw Data migration from a Druid cluster to Redshift and later Snowflake.
- Introduced and rolled out Machine Learning Experiment Tracking System.
- Deployed and maintained first on-prem Kubernetes Clusters in the company.

Senior Data Engineer / Team Lead

- Led a successful implementation and optimisation of a custom Website Activity Tracking Ingestion Pipeline.
- Deployed and rolled out a company-wide Airflow Cluster.
- Led a full migration from Jenkins to Airflow for Data Engineering Pipelines.

Senior Data Engineer

Tesonet

Was responsible for Identifying use cases for Machine Learning and implementing ML Based Products end to end:

- Data Extraction.
- Data cleaning and validation.
- Model training.
- Model Packaging and Deployment.

Participated in designing and building Big Data Ingestion Pipelines:

- Setting up and configuring Kafka Topics.
- Developing Kafka Streams and Spark applications.

Senior Data Scientist/Engineer

Danske Bank

Projects I worked on:

-Affordability model enhancement using transaction data.
-Bringing transaction data into Balance Sheet stress-testing.
-Unlocking potential of transaction data throughout Risk Management function.
-Stress-testing of REA components and dynamic analysis on different credit portfolio levels.
-Process optimization and automation. Preparing models for production placement.

Senior Data Scientist/Analyst

Projects I worked on:

-Retail customer affordability project in Big Data analytics team.
-IFRS9 Life-time probability of default (LTPD) model development.
-IFRS9 Prepayment model development.
-IFRS9 impact on P&L analysis.
-IFRS9 and pre-IFRS9 ECL stress-testing and reporting.
-REA stress-testing and dynamic analysis on different credit portfolio levels.
-Process optimization by both improving old approaches and introducing new technologies.

Analyst

ADB Gjensidige

Was responsible for implementation of Solvency2 regulatory standards and providing required reports.

Actuary

Was responsible for generating business analysis reports for management.

Was responsible for calculation of Technical Reserves for Non-Life insurance products by applying different triangulation methods.

Junior Actuary

Follow me to learn about Data Engineering and ML Systems | Author of SwirlAI Newsletter | Public Speaker

I am a seasoned Data Professional with over a decade experience in Data Science, Data/ML/MLOps Engineering fields. I am obsessed with building out Data Capabilities at scale and helping organisations implement their Data Driven visions. 

I believe that a strong Self Service Data and ML Platform is key for organisations to be successful in scaling out their Data efforts. To achieve it there is both a need of Cultural and Technical transformation.

I am also helping practitioners to Level Up in MLOps, Data Engineering, Machine Learning and related Data fields by sharing my knowledge and experience in easy to understand digestible way.

Leading Product Strategy.

What are 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) 𝗦𝘆𝘀𝘁𝗲𝗺𝘀?

Here is an example of a simple RAG based Chatbot to query your Private Knowledge Base.

First step is to store the knowledge of your internal documents in a format that is suitable for querying. We do so by embedding it using an embedding model:

𝟭: Split text corpus of the entire knowledge base into chunks - a chunk will represent a single piece of context available to be queried. Data of interest can be from multiple sources, e.g. Documentation in Confluence supplemented by PDF reports.
𝟮: Use the Embedding Model to transform each of the chunks into a vector embedding.
𝟯: Store all vector embeddings in a Vector Database.
𝟰: Save text that represents each of the embeddings separately together with the pointer to the embedding (we will need this later).

Next we can start constructing the answer to a question/query of interest:

𝟱: Embed a question/query you want to ask using the same Embedding Model that was used to embed the knowledge base itself.
𝟲: Use the resulting Vector Embedding to run a query against the index in the Vector Database. Choose how many vectors you want to retrieve from the Vector Database - it will equal the amount of context you will be retrieving and eventually using for answering the query question.
𝟳: Vector DB performs an Approximate Nearest Neighbour (ANN) search for the provided vector embedding against the index and returns previously chosen amount of context vectors. The procedure returns vectors that are most similar in a given Embedding/Latent space. 
𝟴: Map the returned Vector Embeddings to the text chunks that represent them.
𝟵: Pass a question together with the retrieved context text chunks to the LLM via prompt. Instruct the LLM to only use the provided context to answer the given question. This does not mean that no Prompt Engineering will be needed - you will want to ensure that the answers returned by LLM fall into expected boundaries, e.g. if there is no data in the retrieved context that could be used make sure that no made up answer is provided.

To make it a real Chatbot - face the entire application with a Web UI that exposes a text input box to act as a chat interface. After running the provided question through steps 1. to 9. - return and display the generated answer. This is how most of the chatbots that are based on a single or multiple internal knowledge base sources are actually built nowadays.

As described, the system is really just a naive RAG that is usually not fit for production grade applications. You need to understand all of the moving pieces in the system in order to tune them by applying advanced techniques, consequently transforming the Naive RAG to Advanced RAG fit for production. More on this in the upcoming posts, so stay tuned in!

#LLM #GenAI #LLMOps #MachineLearning

Chief Product Officer @ neptune.ai | Follow me to Learn about LLM and Data Engineering Systems | Author of SwirlAI Newsletter | Public Speaker

What is a 𝗩𝗲𝗰𝘁𝗼𝗿 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲?

With the rise of Foundational Models, Vector Databases skyrocketed in popularity. The truth is that a Vector Database is also useful outside of a Large Language Model context.

When it comes to Machine Learning, we often deal with Vector Embeddings. Vector Databases were created to perform specifically well when working with them:

➡️ Storing.
➡️ Updating.
➡️ Retrieving.

When we talk about retrieval, we refer to retrieving set of vectors that are most similar to a query in a form of a vector that is embedded in the same Latent space. This retrieval procedure is called Approximate Nearest Neighbour (ANN) search.

A query here could be in a form of an object like an image for which we would like to find similar images. Or it could be a question for which we want to retrieve relevant context that could later be transformed into an answer via a LLM.

Let’s look into how one would interact with a Vector Database:

𝗪𝗿𝗶𝘁𝗶𝗻𝗴/𝗨𝗽𝗱𝗮𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮.

1. Choose a ML model to be used to generate Vector Embeddings.
2. Embed any type of information: text, images, audio, tabular. Choice of ML model used for embedding will depend on the type of data.
3. Get a Vector representation of your data by running it through the Embedding Model.
4. Store additional metadata together with the Vector Embedding. This data would later be used to pre-filter or post-filter ANN search results.
5. Vector DB indexes Vector Embedding and metadata separately. There are multiple methods that can be used for creating vector indexes, some of them: Random Projection, Product Quantization, Locality-sensitive Hashing.
6. Vector data is stored together with indexes for Vector Embeddings and metadata connected to the Embedded objects.

𝗥𝗲𝗮𝗱𝗶𝗻𝗴 𝗗𝗮𝘁𝗮.

7. A query to be executed against a Vector Database will usually consist of two parts:

➡️ Data that will be used for ANN search. e.g. an image for which you want to find similar ones.
➡️ Metadata query to exclude Vectors that hold specific qualities known beforehand. E.g. given that you are looking for similar images of apartments - exclude apartments in a specific location.

8. You execute Metadata Query against the metadata index. It could be done before or after the ANN search procedure.
9. You embed the data into the Latent space with the same model that was used for writing the data to the Vector DB.
10. ANN search procedure is applied and a set of Vector embeddings are retrieved. Popular similarity measures for ANN search include: Cosine Similarity, Euclidean Distance, Dot Product.

Some popular Vector Databases: Qdrant, Pinecone, Weviate, Milvus, Faiss, Vespa.

How are you using Vector DBs? Let me know in the comment section!

#MachineLearning #GenAI #LLM #AI

How is 𝗖𝗜/𝗖𝗗 𝗽𝗿𝗼𝗰𝗲𝘀𝘀 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗳𝗼𝗿 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝘀 compared to 𝗥𝗲𝗴𝘂𝗹𝗮𝗿 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲? 
 
The important difference that the Machine Learning aspect of the projects brings to the CI/CD process is the treatment of the Machine Learning Training pipeline as a first class citizen of the software world. 
 
➡️ CI/CD pipeline is a separate entity from Machine Learning Training pipeline. There are frameworks and tools that provide capabilities specific to Machine Learning pipelining needs (e.g. KubeFlow Pipelines, Sagemaker Pipelines etc.). 
➡️ ML Training pipeline is an artifact produced by Machine Learning project and should be treated in the CI/CD pipelines as such. 
 
What does it mean? Let’s take a closer look: 
 
Regular CI/CD pipelines will usually be composed of at-least three main steps. These are: 
 
𝗦𝘁𝗲𝗽 𝟭: Unit Tests - you test your code so that the functions and methods produce desired results for a set of predefined inputs. 
 
𝗦𝘁𝗲𝗽 𝟮: Integration Tests - you test specific pieces of the code for ability to integrate with systems outside the boundaries of your code (e.g. databases) and between the pieces of the code itself. 
 
𝗦𝘁𝗲𝗽 𝟯: Delivery - you deliver the produced artifact to a pre-prod or prod environment depending on which stage of GitFlow you are in. 
 
What does it look like when ML Training pipelines are involved? 
 
𝗦𝘁𝗲𝗽 𝟭: Unit Tests - in mature MLOps setup the steps in ML Training pipeline should be contained in their own environments and Unit Testable separately as these are just pieces of code composed of methods and functions. 
 
𝗦𝘁𝗲𝗽 𝟮: Integration Tests - you test if ML Training pipeline can successfully integrate with outside systems, this includes connecting to a Feature Store and extracting data from it, ability to hand over the ML Model artifact to the Model Registry, ability to log metadata to ML Metadata Store etc. This CI/CD step also includes testing the integration between each of the Machine Learning Training pipeline steps, e.g. does it succeed in passing validation data from training step to evaluation step. 
 
𝗦𝘁𝗲𝗽 𝟯: Delivery - the pipeline is delivered to a pre-prod or prod environment depending on which stage of GitFlow you are in. If it is a production environment, the pipeline is ready to be used for Continuous Training. You can trigger the training or retraining of your ML Model ad-hoc, periodically or if the deployed model starts showing signs of Feature/Concept Drift. 
 
Let me know your thoughts. 👇 
 
#MachineLearning #GenAI #LLM #LLMOps

No alternative text description for this image

What is the difference between 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗕𝘂𝗰𝗸𝗲𝘁𝗶𝗻𝗴 𝗶𝗻 𝗦𝗽𝗮𝗿𝗸?

When working with big data there are many important concepts we need to consider about how the data is stored both on disk and in memory, we should try to answer questions like:

➡️ Can we achieve desired parallelism?

➡️ Can we skip reading parts of the data?
✅ The question is addressed by partitioning and bucketing procedures

➡️ How is the data colocated on disk?
✅ The question is mostly addressed by bucketing.

So what are the procedures of Partitioning and Bucketing? 𝗟𝗲𝘁'𝘀 𝘇𝗼𝗼𝗺 𝗶𝗻.

𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴.

➡️ Partitioning in Spark API is implemented by .partitionBy() method of the DataFrameWriter class.
➡️ You provide the method one or multiple columns to partition by.
➡️ The dataset is written to disk split by the partitioning column, each of the partitions is saved into a separate folder on disk.
➡️ Each folder can maintain multiple files, the amount of resulting files is controlled by the setting spark.sql.shuffle.partitions.

✅ Partitioning enables Partition Pruning. Given we filter on a column that we used to partition the dataframe by, Spark can plan to skip the reading of files that are not falling into the filter condition.

𝗕𝘂𝗰𝗸𝗲𝘁𝗶𝗻𝗴.

➡️ Bucketing in Spark API is implemented by .bucketBy() method of the DataFrameWriter class.

𝟭: We have to save the dataset as a table since the metadata of buckets has to be saved somewhere. Usually, you will find a Hive metadata store leveraged here.
𝟮: You will need to provide number of buckets you want to create. Bucket number for a given row is assigned by calculating a hash on the bucket column and performing modulo by the number of desired buckets operation on the resulting hash.
𝟯: Rows of a dataset being bucketed are assigned to a specific bucket and collocated when saving to disk.

✅ If Spark performs wide transformation between the two dataframes, it might not need to shuffle the data as it is already collocated in the executors correctly and Spark is able to plan for that.

❗️There are conditions that need to be met between two datasets in order for bucketing to have desired effect.

𝗪𝗵𝗲𝗻 𝘁𝗼 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝘄𝗵𝗲𝗻 𝘁𝗼 𝗕𝘂𝗰𝗸𝗲𝘁?

✅ If you will often perform filtering on a given column and it is of low cardinality, partition on that column.
✅ If you will be performing complex operations like joins, groupBys and windowing and the column is of high cardinality, consider bucketing on that column.

❗️Bucketing is complicated to nail as there are many caveats and nuances you need to know when it comes to it. More on it in future posts.

#DataEngineering #MachineLearning

What does an 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 look like? 
 
MLOps practices are there to improve Machine Learning Product development velocity, the biggest bottlenecks happen when Experimentation Environments and other infrastructure elements are integrated poorly. 
 
Let’s look into the properties that an effective Experimentation Environment should have. As a MLOps engineer you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for. 
 
𝟭: Access to the raw data. While handling raw data is the responsibility of Data Engineering function, Data Scientists need the ability to explore and analyze available raw data and decide which of it needs to be moved upstream the Data Value Chain (2.1). 
 
𝟮: Access to the curated data. Curated data might be available in the Data Warehouse but not exposed via a Feature Store. Such Data should not be exposed for model training in production environments. Data Scientists need the ability to explore curated data and see what needs to be pushed downstream (3.1). 
 
𝟯: Data used for training of Machine Learning models should be sourced from a Feature Store if the ML Training pipeline is ready to be moved to the production stage. 
 
𝟰: Data Scientists should be able to easily spin up different types of compute clusters - might it be Spark, Dask or any other technology - to allow effective Raw and Curated Data exploration. 
 
𝟱: Data Scientists should be able to spin up a production like remote Machine Learning Training pipeline in development environment ad-hoc from the Notebook, this increases speed of iteration significantly. 
 
𝟲: There should be an automated setup in place that would perform the testing and promotion to a higher env when a specific set of Pull Requests are created. E.g. a PR from feature/* to release/* branch could trigger a CI/CD process to test and deploy the ML Pipeline to a pre-prod environment. 
 
𝟳: Notebooks and any additional boilerplate code for CI/CD should be part of your Git integration. Make it crystal clear where a certain type of code should live - a popular way to do this is providing repository templates with clear documentation. 
 
𝟴: Experiment/Model Tracking System should be exposed to both local and remote pipelines. 
 
𝟗: Notebooks have to be running in the same environment that your production code will run in. Incompatible dependencies should not cause problems when porting applications to production. It can be achieved by running Notebooks in containers. 
 
Did I miss something? 👇 
 
#GenAI #LLM #LLMOps #MachineLearning

What is the role of 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗿𝗮𝗰𝘁𝘀 in the Data Pipeline?
 
In its simplest form Data Contract is an agreement between Data Producers and Data Consumers on what the Data being produced should look like, what SLAs it should meet and the semantics of it.
 
𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗿𝗮𝗰𝘁 𝘀𝗵𝗼𝘂𝗹𝗱 𝗵𝗼𝗹𝗱 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝗻𝗼𝗻-𝗲𝘅𝗵𝗮𝘂𝘀𝘁𝗶𝘃𝗲 𝗹𝗶𝘀𝘁 𝗼𝗳 𝗺𝗲𝘁𝗮𝗱𝗮𝘁𝗮:
 
👉 Schema of the Data being Produced.
👉 Schema Version - Data Sources evolve, Producers have to ensure that it is possible to detect and react to schema changes. Consumers should be able to process Data with the old Schema.
👉 SLA metadata - Quality: is it meant for Production use? How late can the data arrive? How many missing values could be expected for certain fields in a given time period?
👉 Semantics - what entity does a given Data Point represent. Semantics, similar to schema, can evolve over time.
👉 Lineage - Data Owners, Intended Consumers.
👉 …
 
𝗦𝗼𝗺𝗲 𝗣𝘂𝗿𝗽𝗼𝘀𝗲𝘀 𝗼𝗳 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗿𝗮𝗰𝘁𝘀:
 
➡️ Ensure Quality of Data in the Downstream Systems.
➡️ Prevent Data Processing Pipelines from unexpected outages.
➡️ Enforce Ownership of produced data closer to where it was generated.
➡️ Improve scalability of your Data Systems.
➡️ Reduce intermediate Data Handover Layer.
➡️ …
 
Example implementation for Data Contract Enforcement:
 
1️⃣ Schema changes are implemented in a git repository, once approved - they are pushed to the Applications generating the Data and a central Schema Registry.
2️⃣ Applications push generated Data to Kafka Topics. Separate Raw Data Topics for CDC streams and Direct emission.
3️⃣ A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Schema Registry.
4️⃣ Data that does not meet the contract is pushed to Dead Letter Topic.
5️⃣ Data that meets the contract is pushed to Validated Data Topic.
6️⃣ Applications that need Real Time Data consume it directly from Validated Data Topic or its derivatives.
7️⃣ Data from the Validated Data Topic is pushed to object storage for additional Validation.
8️⃣ On a schedule Data in the Object Storage is validated against additional SLAs and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes. 
9️⃣ Consumers and Producers are alerted to any SLA breaches.
🔟 Data that was Invalidated in Real Time is consumed by Flink Applications that alert on invalid schemas. There could be a recovery Flink App with logic on how to fix invalidated Data.
 
Leave your thoughts in the comment section 👇

#DataEngineering #MachineLearning

What is 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 (𝗖𝗧) in MLOps and what steps are needed to achieve it?

CT is the process of automated ML Model retraining in Production Environments on a specific trigger. Let’s look into some prerequisites for this:
 
1) Automation of ML Pipelines.
 
- Pipelines are orchestrated.
- Each pipeline step is developed independently and is able to run on different technology stacks.
- Pipelines are treated as a code artifact.
 
✅ You deploy Pipelines instead of Model Artifacts allowing Continuous Training In production.
✅ Reuse of components allows for rapid experimentation.
 
2) Introduction of strict Data and Model Validation steps in the ML Pipeline.
 
- Data is validated before training the Model. If inconsistencies are found - Pipeline is aborted.
- Model is validated after training. Only after it passes the validation is it handed over for deployment.
 
✅ Short circuits of the Pipeline allow for safe CT in production.
 
3) Introduction of ML Metadata Store.
 
- Any Metadata related to ML artifact creation is tracked here.
- We also track performance of the ML Model.
 
✅ Experiments become reproducible and comparable between each other.
✅ Model Registry acts as glue between training and deployment pipelines.
 
4) Different Pipeline triggers in production.
 
- Ad-hoc.
- Cron.
- Reactive to Metrics produced in Model Monitoring System.
- Arrival of New Data.

✅ This is where the Continuous Training is actually triggered.
 
5) Introduction of Feature Store (Optional).
 
- Avoid work duplication when defining features.
- Reduce risk of Training/Serving Skew.
 
𝗠𝘆 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀 𝗼𝗻 𝗖𝗧:
 
➡️ Introduction of CT is not straightforward and you should approach it iteratively. The following could be good Quarterly Goals to set:
 
- Experiment Tracking is extremely important at any level of ML Maturity and the least invasive in the process of ML Model training - I would start with ML Metadata Store introduction.
- Orchestration of ML Pipelines is always a good idea, there are many tools supporting this (Airflow, Kubeflow, VertexAI etc.). If you are not doing it yet - grab this next, also make the validation steps part of this goal.
- The need for Feature Store will wary on the types of Models you are deploying. I would prioritise it if you have Models that perform Online predictions as it will help with avoiding Training/Serving Skew.
- Don’t rush with Automated retraining. Ad-hoc and on-schedule will bring you a long way.
 
Let me know your thoughts! 👇

#LLM #MachineLearning #GenAI

What are the most common 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀 𝗳𝗼𝗿 𝗞𝗮𝗳𝗸𝗮?

We have covered lots of concepts around Kafka already. But what are the most common use cases for The System that you are very likely to run into as a Data Engineer?

𝗟𝗲𝘁’𝘀 𝘁𝗮𝗸𝗲 𝗮 𝗰𝗹𝗼𝘀𝗲𝗿 𝗹𝗼𝗼𝗸:

𝗪𝗲𝗯𝘀𝗶𝘁𝗲 𝗔𝗰𝘁𝗶𝘃𝗶𝘁𝘆 𝗧𝗿𝗮𝗰𝗸𝗶𝗻𝗴.

➡️ The Original use case for Kafka by LinkedIn.
➡️ Events happening in the website like page views, conversions etc. are sent via a Gateway and piped to Kafka Topics.
➡️ These events are forwarded to the downstream Analytical systems or processed in Real Time.
➡️ Kafka is used as an initial buffer as the Data amounts are usually big and Kafka guarantees no message loss due to its replication mechanisms.

𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗥𝗲𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻.

➡️ Database Commit log is piped to a Kafka topic.
➡️ The committed messages are executed against a new Database in the same order.
➡️ Database replica is created.

𝗟𝗼𝗴/𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻.

➡️ Kafka is used for centralized Log and Metrics collection.
➡️ Daemons like FluentD are deployed in servers or containers together with the Applications to be monitored.
➡️ Applications send their Logs/Metrics to the Daemons.
➡️ The Daemons pipe Logs/Metrics to a Kafka Topic.
➡️ Logs/Metrics are delivered downstream to storages like ElasticSearch or InfluxDB for Log/Metrics discovery respectively.
➡️ This is also how you would track your IoT Fleets.

𝗦𝘁𝗿𝗲𝗮𝗺 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴.

➡️ This is usually coupled with ingestion mechanisms already covered.
➡️ Instead of piping Data to a certain storage downstream we mount a Stream Processing Framework on top of Kafka Topics.
➡️ The Data is filtered, enriched and then piped to the downstream systems to be further used according to the use case.
➡️ This is also where one would be running Machine Learning Models embedded into a Stream Processing Application.

𝗠𝗲𝘀𝘀𝗮𝗴𝗶𝗻𝗴.

➡️ Kafka can be used as a replacement for more traditional messaging brokers like RabbitMQ.
➡️ Kafka has better durability guarantees and is easier to configure for several separate Consumer Groups to consume from the same Topic.

❗️Having said this - always consider the complexity you are bringing with introduction of a Distributed System. Sometimes it is better to just use traditional frameworks.

#DataEngineering #Data #MachineLearning

What are the four main 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗠𝗼𝗱𝗲𝗹 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝗧𝘆𝗽𝗲𝘀?
 
Even if you will not work with them day to day, the following are the four ways to deploy a ML Model you should know and understand as a MLOps/ML Engineer.
 
➡️ 𝗕𝗮𝘁𝗰𝗵: 
 
👉 You apply your trained models as a part of ETL/ELT Process on a given schedule.
👉 You load the required Features from a batch storage, apply inference and save the results to a batch storage.
👉 It is sometimes falsely thought that you can’t use this method for Real Time Predictions.
👉 Inference results can be loaded into a real time storage and used for real time applications.
 
➡️ 𝗘𝗺𝗯𝗲𝗱𝗱𝗲𝗱 𝗶𝗻 𝗮 𝗦𝘁𝗿𝗲𝗮𝗺 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻: 
 
👉 You apply your trained models as a part of Stream Processing Pipeline.
👉 While Data is continuously piped through your Streaming Data Pipelines, an application with a loaded model continuously applies inference on the data and returns it to the system - most likely another Streaming Storage.
👉 This deployment type is likely to involve a real time Feature Store Serving API to retrieve additional Static Features for inference purposes.
👉 Predictions can be consumed by multiple applications subscribing to the Inference Stream.
 
➡️ 𝗥𝗲𝗮𝗹 𝗧𝗶𝗺𝗲:
 
👉 You expose your model as a Backend Service (REST or gRPC).
👉 This ML Service retrieves Features needed for inference from a Real Time Feature Store Serving API.
👉 Inference can be requested by any application in real time as long as it is able to form a correct request that conforms API Contract.
 
➡️ 𝗘𝗱𝗴𝗲: 
 
👉 You embed your trained model directly into the application that runs on a user device.
👉 This method provides the lowest latency and improves privacy.
👉 Data in most cases is generated and lives inside of device significantly improving the security.
 
What types of deployments are you mostly working on? Let me know in the comments! 👇

#GenAI #LLM #MachineLearning

Building a 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗴𝗿𝗮𝗱𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) 𝗯𝗮𝘀𝗲𝗱 𝗟𝗟𝗠 𝘀𝘆𝘀𝘁𝗲𝗺 is more than just writing few lines of code using LangChain.

Here are some of the moving parts in the RAG based systems that you will need to take care of and continuously tune in order to achieve desired results:

𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹:

𝘍 ) Chunking - how do you chunk the data that you will use for external context.

 - Small, Large chunks.
 - Sliding or tumbling window for chunking.
 - Retrieve parent or linked chunks when searching or just use originally retrieved data.

𝘊 ) Choosing the embedding model to embed and query external context to/from the latent space.

𝘋 ) Vector Database.

 - Which Database to choose.
 - Where to host.
 - What metadata to store together with embeddings.
 - Indexing strategy.

𝘌 ) Vector Search

 - Choice of similarity measure.
 - Choosing the query path - metadata first vs. ANN first.
 - Hybrid search.

𝘎 ) Heuristics - business rules applied to your retrieval procedure.

 - Time importance.
 - Reranking.
 - Duplicate context (diversity ranking).
 - Source retrieval.
 - Conditional document preprocessing.

𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻:

𝘈 ) LLM - Choosing the right Large Language Model to power your application.

✅ It is becoming less of a headache the further we are into the LLM craze. The performance of available LLMs are converging, both open source and proprietary. The main choice nowadays is around using a proprietary model or self-hosting.

𝘉 ) Prompt Engineering - having context available for usage in your prompts does not free you from the hard work of engineering the prompts. You will still need to align the system to produce outputs that you desire and prevent jailbreak scenarios by Guard-railing inputs and outputs.

What other pieces of the system am I missing? Let me know in the comments 👇

#LLM #GenAI #LLMOps #MachineLearning

Ensuring 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗶𝗻 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 is now as important as ever, how do we do it?

It is extremely important to ensure Data Quality upstream of ML Training and Inference Pipelines, trying to do that in the pipelines will cause unavoidable failure when working at scale. Data Contracts can be leveraged for this goal.
 
Data Contract is an agreement between Data Producers and Data Consumers about the qualities to be met by Data being produced.

Data Contract should hold the following non-exhaustive list of metadata:
 
👉 Schema Definition.
👉 Schema Version.
👉 SLA metadata.
👉 Semantics.
👉 Lineage.
👉 …

Example Architecture Enforcing Data Contracts:
 
𝟭: Schema changes are implemented in version control, once approved - they are pushed to the Applications generating the Data, Databases holding the Data and a central Data Contract Registry.

[𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁]: Ideally you should be enforcing a Data contract at this stage, when producing Data. Data Validation steps down the stream are Detection and Prevention mechanisms that don’t allow low quality data to reach downstream systems. There might be a significant delay before you can do those checks, causing irreversible corruption or loss of data.
 
Applications push generated Data to Kafka Topics:
 
𝟮: Events emitted directly by the Application Services.
 
👉 This also includes IoT Fleets and Website Activity Tracking.
 
𝟮.𝟭: Raw Data Topics for CDC streams.
 
𝟯: A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Contract Registry.
𝟰: Data that does not meet the contract is pushed to Dead Letter Topic.
𝟱: Data that meets the contract is pushed to Validated Data Topic.
𝟲: Data from the Validated Data Topic is pushed to object storage for additional Validation.
𝟳: On a schedule Data in the Object Storage is validated against additional SLAs in Data Contracts and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes.
𝟴: Modeled and Curated data is pushed to the Feature Store System for further Feature Engineering.
𝟴.𝟭: Real Time Features are ingested into the Feature Store directly from Validated Data Topic (5).
 
👉 Ensuring Data Quality here is complicated since checks against SLAs is hard to perform.
 
𝟵: High Quality Data is used in Machine Learning Training Pipelines.
𝟭𝟬: The same Data is used for Feature Serving in Inference.

Note: ML Systems are plagued by other Data related issues like Data and Concept Drifts. These are silent failures and while they can be monitored, we don’t include it in the Data Contract.

It is great to see a conference out there that is dedicated to Quality of AI Systems - AIQCon! If you happen to be in San Francisco on June 25th, be sure to drop by.

Great line up of speakers, check it out here: https://lnkd.in/dYR-GPyc

Let me know your thoughts! 👇

#MachineLearning #DataEnginering #GenAI

Why is 𝗠𝗼𝗱𝗲𝗹 𝗥𝗲𝗴𝗶𝘀𝘁𝗿𝘆 so important in your MLOps Stack?
 
Let's look again into the Machine Learning Model Training Lifecycle.
 
𝗟𝗲𝘁’𝘀 𝗿𝗲𝘃𝗶𝗲𝘄 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝘀𝘁𝗲𝗽𝘀:
 
1: Version Control: Machine Learning Training Pipeline is defined in code, once merged to the main branch it is built and triggered.
2: Feature Preprocessing: Features are retrieved from the Feature Store, validated and passed to the next stage. Any feature related metadata that is tightly coupled to the Model being trained is saved to the Experiment Tracking System.
3: Model is trained and validated on Preprocessed Data, Model related metadata is saved to the Experiment Tracking System.
4: If Model Validation passes all checks - Model Artifact is passed to a Model Registry. The model is stored and ready to be packaged.
 
𝗧𝗵𝗶𝘀 𝗶𝘀 𝗶𝘁 - regardless of what deployment type will follow, the model is stored in The Model Registry. Model Registry is what glues Training and Deployment Pipelines together and this is where handover of the Model Artifact happens. 
 
5: The same model can be packaged as a container for different deployment types by implementing a respective interface. E.g.
 
- Flink application for Stream Processing Deployment.
- gRPC API for Request-Response.
- Plain model pointing to the Batch Serving Feature Store API for Batch.
 
- 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗴𝗼𝗼𝗱 𝗻𝗲𝘄𝘀 - as long as you are training your models using historical data the training pipeline is the same for any type of deployment.
- It’s a different story when it comes to online training!
 
Now this is a bit different in the era of Large Language Models, but more on it in the upcoming posts. Stay tuned in!
 
Leave any thoughts in the comment section 👇

#GenAI #LLM #MachineLearning

What is the difference between 𝗟𝗮𝗺𝗯𝗱𝗮 𝗮𝗻𝗱 𝗞𝗮𝗽𝗽𝗮 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀?
 
Lambda and Kappa are both Data architectures proposed to solve movement of large amounts of data for reliable Online access.
 
The most popular architecture has been and continues to be Lambda. However, with Stream Processing becoming more accessible you will be hearing a lot more of Kappa in the near future. Let’s see how they are different.
 
𝗟𝗮𝗺𝗯𝗱𝗮.
 
➡️ Ingestion layer is responsible for collecting the raw data and duplicating it for further Real Time and Batch processing separately.
➡️ Consists of 3 additional main layers:
 
👉 Speed or Stream - Raw Data here is coming in Real Time and is processed by a Stream Processing Framework (e.g. Flink) then passed to the Serving layer to create Real Time Views for low latency near to Real Time Data access.
👉 Batch - Batch ETL Jobs with batch processing Frameworks (e.g. Spark) are run against raw Data to create reliable Batch Views for Offline Historical Data access.
👉 Serving - this is where the processed Data is exposed to the end user. Latest Real Time Data can be accessed from Real Time Views or combined with Batch Views for full history. Historical Data can be accessed from Batch Views.
 
❗️ Processing code is duplicated for different technologies in Batch and Speed Layers causing logic divergence.
❗️ Compute resources are duplicated.
❗️ Need to manage two Infrastructures.
✅ Distributed Batch Storage is reliable and scalable, even if the System crashes it is easily recoverable without errors.
 
𝗞𝗮𝗽𝗽𝗮.
 
➡️ Treats both Batch and Real Time Workloads as a Stream Processing problem.
➡️ Uses Speed Layer only to prepare data for Real Time and Batch Access.
➡️ Consists of only 2 main layers:
 
👉 Speed or Stream - similar to Lambda but (optionally) often contains Tiered Storage which means that all of Data coming into the system is stored indefinitely in different Storage Layers. E.g. S3 or GCS for historical data and on disk log for hot data.
👉 Serving - same as Lambda but all transformations that are performed in Speed Layer are never duplicated in Batch Layer.
 
❗️ Some transformations are hard to perform in Speed Layer (e.g. complex joins) and are eventually pushed to Batch storage for implementation.
❗️ Requires strong skills in Stream Processing.
✅ Data is processed once with a single Stream Processing Engine.
✅ Only need to manage single set of Infrastructure.
 
Have you dealt with Kappa architecture in your day-to-day? What are your thoughts around it? Let me know in the comments 👇

#DataEngineering #Data

What is 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 and why you might need it?

When you deploy Machine Learning models to production you need to take into account several operational metrics that are in general not ML related. Today we talk about two of them:

👉 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗟𝗮𝘁𝗲𝗻𝗰𝘆: How long does it take for your Model to compute inference result and return it.
👉 𝗠𝗼𝗱𝗲𝗹 𝗦𝗶𝘇𝗲: How much memory does your model occupy when it’s loaded for serving inference results.

Both of these are important when considering operational performance and feasibility of your model deployment in production.

👉 Large models might not fit on a device if you are considering edge deployments.
👉 Latency of retrieving inference results might make business case non feasible. E.g. Recommendation Engines require latencies in milliseconds as ranking has to be applied as the user browses your website or app in real time.
👉 …

You can influence both latency and size by applying different Model Compression methods, some of them are:

➡️ 𝗣𝗿𝘂𝗻𝗶𝗻𝗴: this method is mostly used in tree-based and Neural Network algorithms. In tree-based ones we prune leaves or branches from decision trees. In Neural Networks we remove nodes and synapses (weights) while trying to retain ML performance metrics.

✅ In both cases the output is a reduction in the number of Model Parameters and model size.

➡️ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗗𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻: this type of compression is achieved by:

👉 Training an original large model which is called the Teacher model. 
👉 Training a smaller model to mimic the Teacher model by transferring knowledge from it, this model is called the Student model. Knowledge in this context can be extracted from the outputs, internal hidden state (feature representations) or a combination of both.
👉 We then use the “Student” model in production.

➡️ 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻: a most commonly used method that doesn’t have much to do with Machine Learning. This approach uses fewer bits to represent model parameters.

👉 You can apply quantization techniques both during the training and after the models has been already trained.
👉 In regular Neural Networks what is quantized are Model Weights, Biases and Activation Functions.
👉 Most usual quantization is from float to integer (32 bits to 8 bits).

➡️ …

[𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁]: while the above methods do reduce the size of the models, allowing them to be deployed in production scenarios, there is almost always a reduction in accuracy so be careful and evaluate it accordingly.

What methods for reducing model size have you used? What were the main challenges? Let me know in the comment section! 

#MachineLearning #GenAI #LLM #AI

Last week I reached the milestone of 𝟭𝟬𝟬.𝟬𝟬𝟬 𝗳𝗼𝗹𝗹𝗼𝘄𝗲𝗿𝘀 𝗼𝗻 𝗟𝗶𝗻𝗸𝗲𝗱𝗜𝗻 🎉

I don’t do many personal posts but these occasions are special. So much has changed in my life in the past year and it is a perfect opportunity to reintroduce myself.

𝗦𝗼𝗺𝗲 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹 𝗳𝗮𝗰𝘁𝘀:

❤️ Recently (almost 2 months ago now) I became a proud father to my beautiful daughter Sofija.
❤️❤️ I am happily married to the love of my life Karolina.
🐶 We are dog people. Nasa the Labrador and Freddie the Poodle are part of our daily lives. In the 50.000 followers update I promised to share a photo with Freddie so here it is 👇
💪 Still trying to return to exercising after Covid hit 😅
📖 I am an avid reader, most of it is technical and leadership related literature, but now and then I love to grab a fantasy novel.

𝗦𝗼𝗺𝗲 𝗽𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 𝗳𝗮𝗰𝘁𝘀:

📊 I have been dealing with data systems for the last 12 years.
🏢 I Have taken roles of: Actuary, Data Scientist, Machine Learning Engineer, Data Engineer, Cloud/Platform Engineer, MLOps Engineer. Also, for several years led teams of different combinations of previously mentioned roles.
🔨 Recently stepped in as Chief Product Officer in ML Developer Tool SaaS startup. So now I am also officially a builder.
🔨 I Have started my own venture (SwirlAI) where I help move the data world forward by educating and providing consulting services.
✏️ I have now been creating content about the data world for more than 2 years.
🗣 2 years ago started my public speaking journey.

𝗪𝗵𝗮𝘁 𝘁𝗼 𝗲𝘅𝗽𝗲𝗰𝘁 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲:

✏️ I will continue sharing my knowledge (existing and newly gained) daily.
🔨 As we build the product, I want to do it in public and share how I think and why I shape the product this way.
👷 More hands-on material.
🤖 A lot more content on LLM based AI systems is coming!

If this excites you, give me a follow!

#GenAI #LLM #MachineLearning #AI

Taplio

Aurimas Griciūnas's Linkedin Analytics