Aurimas Griciūnas

Senior Solutions Architect

SwirlAI

Driving MLOps, Data Engineering and Machine Learning industry forward by helping professionals Level Up in these disciplines.

Founder & CEO

neptune.ai

Chief Product Officer

Head of Product

Driving MLOps industry forward by helping Neptune customers succeed in adopting MLOps best practices.

Transactie Monitoring Nederland (TMNL)

- Contributed to the development of end-to-end Machine Learning platform capabilities in a highly secure environment.
 - Focused on leveraging AWS SageMaker and supporting AWS services to enable scalable experimentation and model deployment.
 - Collaborated with the team to build reusable template libraries, streamlining the process of running ML experiments and deploying models efficiently and securely.

Senior MLOps Engineer

HomeToGo

- Led cross-functional teams of Cloud Engineers and Data Platform Engineers, driving key infrastructure and data initiatives.
 - Spearheaded a company-wide Cloud-Native Transformation initiative to modernize applications and infrastructure.
 - Championed the migration to Infrastructure as Code (IaC), standardizing infrastructure provisioning using Terraform for improved scalability, consistency, and automation.

Data Architect / Data Platform & Cloud Engineering Lead

- Led a Team of Data Engineers.
- Led a Team of Data Platform Engineers.
- Initiated and led a successful company wide Cloud Native Transformation.
- Migrated entire Data Platform Stack to Kubernetes. This included:

        - Airflow.
        - Superset.
        - Spark.
        - MLFlow.
        - Flink.
        - Airbyte.
        - DataHub.

- Introduced first Stream Processing applications that moved some of the Data Warehouse Processing stages upstream the pipeline.

Senior Data Engineer / Data Platform Team Lead

- Led a team of Data Engineers.
- Initiated and led a Data Warehouse migration from Redshift to Snowflake effort. It also included a refactoring of the Data Model.
- Oversaw Data migration from a Druid cluster to Redshift and later Snowflake.
- Introduced and rolled out Machine Learning Experiment Tracking System.
- Deployed and maintained first on-prem Kubernetes Clusters in the company.

Senior Data Engineer / Team Lead

- Led a successful implementation and optimisation of a custom Website Activity Tracking Ingestion Pipeline.
- Deployed and rolled out a company-wide Airflow Cluster.
- Led a full migration from Jenkins to Airflow for Data Engineering Pipelines.

Senior Data Engineer

Tesonet

Responsible for end-to-end development of ML-based products and big data solutions:

 - Identified business use cases suitable for machine learning and translated them into technical solutions.
 - Designed and implemented the full ML lifecycle, including data extraction, cleaning, validation, model training, packaging, and deployment.
 - Contributed to the architecture and development of scalable big data ingestion pipelines.
 - Set up and configured Kafka topics; developed real-time processing applications using Kafka Streams and Spark.

Senior Data Scientist/Engineer

Danske Bank

Projects I worked on:

-Affordability model enhancement using transaction data.
-Bringing transaction data into Balance Sheet stress-testing.
-Unlocking potential of transaction data throughout Risk Management function.
-Stress-testing of REA components and dynamic analysis on different credit portfolio levels.
-Process optimization and automation. Preparing models for production placement.

Senior Data Scientist/Analyst

Projects I worked on:

-Retail customer affordability project in Big Data analytics team.
-IFRS9 Life-time probability of default (LTPD) model development.
-IFRS9 Prepayment model development.
-IFRS9 impact on P&L analysis.
-IFRS9 and pre-IFRS9 ECL stress-testing and reporting.
-REA stress-testing and dynamic analysis on different credit portfolio levels.
-Process optimization by both improving old approaches and introducing new technologies.

Analyst

ADB Gjensidige

Was responsible for implementation of Solvency2 regulatory standards and providing required reports.

Actuary

Was responsible for generating business analysis reports for management.

Was responsible for calculation of Technical Reserves for Non-Life insurance products by applying different triangulation methods.

Junior Actuary

Follow me to learn about Data Engineering and ML Systems | Author of SwirlAI Newsletter | Public Speaker

Tired of all the noise surrounding AI space in recent years? Having a hard time keeping up with all of the news in the field and distinguishing what is really important? Follow me and subscribe to my newsletter to get actionable insights on building production ready AI Systems and keep track of only the important signal of what is happening in the AI world.

With over a decade of experience in Data Science, AI/ML Engineering, and MLOps, I’m also dedicated to helping practitioners level up their skills in MLOps, Data Engineering, Machine Learning, and related fields. I share my knowledge in a clear and accessible way to make learning straightforward and actionable.

If you want to partner with SwirlAI and get your product in front of 40K+ AI/ML/Data Engineers and Leaders via our Newsletter or 135K+ AI/ML/Data Engineers and Leaders via LinkedIn, contact me directly.

Leading Product Strategy.

Integrating 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 Systems via 𝗠𝗖𝗣 👇

If you are building RAG systems and packing many data sources for retrieval, most likely there is some agency present at least at the data source selection for retrieval stage.

This is how MCP enriches the evolution of your Agentic RAG systems in such case (𝘱𝘰𝘪𝘯𝘵 2.):

𝟭. Analysis of the user query: we pass the original user query to a LLM based Agent for analysis. This is where:

➡️ The original query can be rewritten, sometimes multiple times to create either a single or multiple queries to be passed down the pipeline.
➡️ The agent decides if additional data sources are required to answer the query.

𝟮. If additional data is required, the Retrieval step is triggered. We could tap into variety of data types, few examples:

➡️ Real time user data.
➡️ Internal documents that a user might be interested in.
➡️ Data available on the web.
➡️ …

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝗠𝗖𝗣 𝗰𝗼𝗺𝗲𝘀 𝗶𝗻:

✅ Each data domain can manage their own MCP Servers. Exposing specific rules of how the data should be used.
✅ Security and compliance can be ensured on the Servel level for each domain.
✅ New data domains can be easily added to the MCP server pool in a standardised way with no Agent rewrite needed enabling decoupled evolution of the system in terms of 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹, 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗮𝗻𝗱 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆.
✅ Platform builders can expose their data in a standardised way to external consumers. Enabling easy access to data on the web.
✅ AI Engineers can continue to focus on the topology of the Agent.

𝟯. Retrieved data is consolidated and Reranked by a more powerful model compared to regular embedder. Data points are significantly narrowed down.
𝟰. If there is no need for additional data, we try to compose the answer (or multiple answers or a set of actions) straight via an LLM.
𝟱. The answer gets analyzed, summarized and evaluated for correctness and relevance:

➡️ If the Agent decides that the answer is good enough, it gets returned to the user.
➡️ If the Agent decides that the answer needs improvement, we try to rewrite the user query and repeat the generation loop.

Are you using MCP in your Agentic RAG systems? Let me know about your experience in the comment section 👇

Founder @ SwirlAI • UpSkilling the Next Generation of AI Talent • Author of SwirlAI Newsletter • Public Speaker

TOON (Token-Oriented Object Notation) is out for some days now and it aims to make communication with LLMs more accurate and token-efficient.

The TOON topic is now one of the hottest news on the LLM market and it might actually matter.

𝗪𝗵𝘆 𝗜 𝘁𝗵𝗶𝗻𝗸 𝘀𝗼:

I was initially hesitant to cover this, potentially being another hype to quickly fade, but:

✅ The format has been shown to increase the accuracy of models while decreasing the token count. I was not sure if there were any accuracy retention studies made, it seems there were.
✅ Token efficiency is extremely important when working with Agentic Systems that require a lot of structured context inside of their reasoning chains. And we are moving towards a post-PoC world where there is a lot of emphasis placed on optimisation of the workflows.

𝗔 𝘀𝗵𝗼𝗿𝘁 𝘀𝘂𝗺𝗺𝗮𝗿𝘆:

- Token-efficient: typically 30-60% fewer tokens on large uniform arrays vs formatted JSON.
- LLM-friendly guardrails: explicit lengths and fields enable validation.
- Minimal syntax: removes redundant punctuation (braces, brackets, most quotes).
- Indentation-based structure: like YAML, uses whitespace instead of braces.
- Tabular arrays: declare keys once, stream data as rows.

An example:

𝘑𝘚𝘖𝘕 𝘧𝘰𝘳𝘮𝘢𝘵:

"shopping_cart": [
    { "id": "GDKVEG984", "name": "iPhone 15 Pro Max", "quantity": 2, "price": 1499.99, "category": "Electronics" },
    { "id": "GDKVEG985", "name": "Samsung Galaxy S24 Ultra", "quantity": 1, "price": 1299.99, "category": "Electronics" },
    { "id": "GDKVEG986", "name": "Apple Watch Series 9", "quantity": 1, "price": 199.99, "category": "Electronics" },
    { "id": "GDKVEG987", "name": "MacBook Pro 16-inch", "quantity": 1, "price": 2499.99, "category": "Electronics" }
  ]
}

𝘞𝘩𝘦𝘯 𝘦𝘯𝘤𝘰𝘥𝘦𝘥 𝘪𝘯𝘵𝘰 𝘛𝘖𝘖𝘕 𝘧𝘰𝘳𝘮𝘢𝘵:

shopping_cart: items[4]{id,name,quantity,price,category}:
    GDKVEG984,iPhone 15 Pro Max,2,1499.99,Electronics
    GDKVEG985,Samsung Galaxy S24 Ultra,1,1299.99,Electronics
    GDKVEG986,Apple Watch Series 9,1,199.99,Electronics
    GDKVEG987,MacBook Pro 16-inch,1,2499.99,Electronics

𝗥𝗲𝘀𝘂𝗹𝘁:

✅ 43% savings in token amount.
✅ Directly translates to 43% savings in token cost for this LLM input.

❗️ Be sure to know when NOT to use the format (and always test it for your application specifically):

- Deeply nested or non-uniform structures.
- Semi-uniform arrays.
- Pure tabular data.

ℹ️ I will be testing it in the upcoming weeks.

Let me know if you have already tested TOON and what are your takeaways! 👇

A breakdown of 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗶𝗻 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 👇 And yes, it can also be used for LLM based systems!

It is critical to ensure Data Quality and Integrity upstream of ML Training and Inference Pipelines, trying to do that in the downstream systems will cause unavoidable failure when working at scale.

There is a ton of work to be done on the Data Lake or LakeHouse layer. 𝗦𝗲𝗲 𝘁𝗵𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗯𝗲𝗹𝗼𝘄.

𝘌𝘹𝘢𝘮𝘱𝘭𝘦 𝘢𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵𝘶𝘳𝘦 𝘧𝘰𝘳 𝘢 𝘱𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯 𝘨𝘳𝘢𝘥𝘦 𝘦𝘯𝘥-𝘵𝘰-𝘦𝘯𝘥 𝘥𝘢𝘵𝘢 𝘧𝘭𝘰𝘸:

𝟭: Schema changes are implemented in version control, once approved - they are pushed to the Applications generating the Data, Databases holding the Data and a central Data Contract Registry.

Applications push generated Data to Kafka Topics:

𝟮: Events emitted directly by the Application Services.

👉 This also includes IoT Fleets and Website Activity Tracking.

𝟮.𝟭: Raw Data Topics for CDC streams.

𝟯: A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Contract Registry.
𝟰: Data that does not meet the contract is pushed to Dead Letter Topic.
𝟱: Data that meets the contract is pushed to Validated Data Topic.
𝟲: Data from the Validated Data Topic is pushed to object storage for additional Validation.
𝟳: On a schedule Data in the Object Storage is validated against additional SLAs in Data Contracts and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes.
𝟴: Modeled and Curated data is pushed to the Feature Store System for further Feature Engineering.
𝟴.𝟭: Real Time Features are ingested into the Feature Store directly from Validated Data Topic (5).

👉 Ensuring Data Quality here is complicated since checks against SLAs is hard to perform.

𝟵: High Quality Data is used in Machine Learning Training Pipelines.
𝟭𝟬: The same Data is used for Feature Serving in Inference.

Note: ML Systems are plagued by other Data related issues like Data and Concept Drifts. These are silent failures and while they can be monitored, we can’t include it in the Data Contract.

Let me know your thoughts! 👇

Bansari Kikani

𝗠𝗖𝗣 plus 𝗔𝟮𝗔, here is how they complement each other 👇

Protocol wars continue to rage, let's understand how Googles A2A (Agent2Agent) protocol is different from MCP and how they complement each other (read till the end).

𝘔𝘰𝘷𝘪𝘯𝘨 𝘱𝘪𝘦𝘤𝘦𝘴 𝘪𝘯 𝘔𝘊𝘗:

𝟭. MCP Host - Programs using LLMs at the core that want to access data through MCP.

❗️ When combined with A2A, an Agent becomes MCP Host.

𝟮. MCP Client - Clients that maintain 1:1 connections with servers.
𝟯. MCP Server - Lightweight programs that each expose specific capabilities through the standardised Model Context Protocol.
𝟰. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.
𝟱. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

𝘌𝘯𝘵𝘦𝘳 𝘈2𝘈:

Where MCP falls short, A2A tries to help. In multi-Agent applications where state is not necessarily shared

𝟲. Agents (MCP Hosts) would implement and communicate via A2A protocol, that enables:

➡️ Secure Collaboration.
➡️ Task and State Management.
➡️ User Experience Negotiation.
➡️ Capability discovery - similar to MCP tools.

𝗛𝗼𝗻𝗲𝘀𝘁 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ Open protocols for Agent communication are important but they are just a piece of the picture, there will be a need for standards that govern all of the existing protocols and other missing pieces. E.g.

❓ How do we standardise tracing and Observability in multi-agent IOA systems?
❓ How do we retain the identity of the running job of an AI Agent instance if the communication standard is not unified in different parts of the pipeline?
❓ ... 

✅ More on this in my future posts, stay tuned!

Let me know your thoughts in the comments. 👇

I have been developing Agentic Systems for the past few years and the same patterns keep emerging. 👇

𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗗𝗿𝗶𝘃𝗲𝗻 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 is the most reliable way to be successful in building your 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 - here is my template.

Let’s zoom in:

𝟭. Define a problem you want to solve: is GenAI even needed?
𝟮. Build a Prototype: figure out if the solution is feasible.
𝟯. Define Performance Metrics: you must have output metrics defined for how you will measure success of your application.
𝟰. Define Evals: split the above into smaller input metrics that can move the key metrics forward. Decompose them into tasks that could be automated and move the given input metrics. Define Evals for each. Store the Evals in your Observability Platform.

ℹ️ Steps 𝟭. - 𝟰. are where AI Product Managers can help, but can also be handled by AI Engineers.

𝟱. Build a PoC: it can be simple (excel sheet) or more complex (user facing UI). Regardless of what it is, expose it to the users for feedback as soon as possible.
𝟲. Instrument your application: gather traces and human feedback and store it in an Observability Platform next to previously stored Evals.
𝟳. Run Evals on traced data: traces contain inputs and outputs of your application, run evals on top of them.
𝟴. Analyse Failing Evals and negative user feedback: this data is gold as it specifically pinpoints where the Agentic System needs improvement.
𝟵. Use data from the previous step to improve your application - prompt engineer, improve AI system topology, finetune models etc. Make sure that the changes move Evals into the right direction.
𝟭𝟬. Build and expose the improved application to the users.
𝟭𝟭. Monitor the application in production: this comes out of the box - you have implemented evaluations and traces for development purposes, they can be reused for monitoring. Configure specific alerting thresholds and enjoy the peace of mind. 

✅ 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗼𝗳 𝘆𝗼𝘂𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻:

➡️ Run steps 𝟲. - 𝟭𝟬. to continuously improve and evolve your application.
➡️ As you build up in complexity, new requirements can be added to the same application, this includes running steps 𝟭. - 𝟱. and attaching the new logic as routes to your Agentic System.
➡️ You start off with a simple Chatbot and add a route that can classify user intent to take action (e.g. add items to a shopping cart).

What is your experience in evolving Agentic Systems? Let me know in the comments 👇

𝗔𝗜 𝗔𝗴𝗲𝗻𝘁’𝘀 𝗠𝗲𝗺𝗼𝗿𝘆 is the most important piece of 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴, this is how we define it 👇

In general, the memory for an agent is something that we provide via context in the prompt passed to LLM that helps the agent to better plan and react given past interactions or data not immediately available.

It is useful to group the memory into four types:

𝟭. 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 - This type of memory contains past interactions and actions performed by the agent. After an action is taken, the application controlling the agent would store the action in some kind of persistent storage so that it can be retrieved later if needed. A good example would be using a vector Database to store semantic meaning of the interactions.
𝟮. 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 - Any external information that is available to the agent and any knowledge the agent should have about itself. You can think of this as a context similar to one used in RAG applications. It can be internal knowledge only available to the agent or a grounding context to isolate part of the internet scale data for more accurate answers.
𝟯. 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹 - This is systemic information like the structure of the System Prompt, available tools, guardrails etc. It will usually be stored in Git, Prompt and Tool Registries.
𝟰. Occasionally, the agent application would pull information from long-term memory and store it locally if it is needed for the task at hand.
𝟱. All of the information pulled together from the long-term or stored in local memory is called short-term or working memory. Compiling all of it into a prompt will produce the prompt to be passed to the LLM and it will provide further actions to be taken by the system.

We usually label 1. - 3. as Long-Term memory and 5. as Short-Term memory.

And that is it! The rest is all about how you architect the topology of your Agentic Systems.

Any war stories you have while managing Agent’s memory? Let me know in the comments 👇

#AI #LLM

You must know these 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗦𝘆𝘀𝘁𝗲𝗺 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗣𝗮𝘁𝘁𝗲𝗿𝗻𝘀 as an 𝗔𝗜 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿.

If you are building Agentic Systems in an Enterprise setting you will soon discover that the simplest workflow patterns work the best and bring the most business value.

At the end of last year Anthropic did a great job summarising the top patterns for these workflows and they still hold strong.

Let’s explore what they are and where each can be useful:

𝟭. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗖𝗵𝗮𝗶𝗻𝗶𝗻𝗴: This pattern decomposes a complex task and tries to solve it in manageable pieces by chaining them together. Output of one LLM call becomes an output to another.

✅ In most cases such decomposition results in higher accuracy with sacrifice for latency.
ℹ️ In heavy production use cases Prompt Chaining would be combined with following patterns, a pattern replace an LLM Call node in Prompt Chaining pattern.

𝟮. 𝗥𝗼𝘂𝘁𝗶𝗻𝗴: In this pattern, the input is classified into multiple potential paths and the appropriate is taken.

✅ Useful when the workflow is complex and specific topology paths could be more efficiently solved by a specialized workflow.
ℹ️ Example: Agentic Chatbot - should I answer the question with RAG or should I perform some actions that a user has prompted for?

𝟯. 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Initial input is split into multiple queries to be passed to the LLM, then the answers are aggregated to produce the final answer.

✅ Useful when speed is important and multiple inputs can be processed in parallel without needing to wait for other outputs. Also, when additional accuracy is required.
ℹ️ Example 1: Query rewrite in Agentic RAG to produce multiple different queries for majority voting. Improves accuracy.
ℹ️ Example 2: Multiple items are extracted from an invoice, all of them can be processed further in parallel for better speed.

𝟰. 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗼𝗿: An orchestrator LLM dynamically breaks down tasks and delegates to other LLMs or sub-workflows.

✅ Useful when the system is complex and there is no clear hardcoded topology path to achieve the final result. 
ℹ️ Example: Choice of datasets to be used in Agentic RAG.

𝟱. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿-𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗿: Generator LLM produces a result then Evaluator LLM evaluates it and provides feedback for further improvement if necessary.

✅ Useful for tasks that require continuous refinement.
ℹ️ Example: Deep Research Agent workflow when refinement of a report paragraph via continuous web search is required.

𝗧𝗶𝗽𝘀:

❗️ Before going for full fledged Agents you should always try to solve a problem with simpler Workflows described in the article.

What are the most complex workflows you have deployed to production? Let me know in the comments 👇

#LLM #AI #MachineLearning

𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 is a must have in your tool belt as an AI Engineer. 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 sits at the core of it, why is it important?

Tracing and instrumentation of software have been around for decades now. With AI systems resembling regular software even more, we are now moving the practice here as well (with a few key differences).

Let’s look into the process of tracing from a perspective of a naive RAG system.

𝘍𝘦𝘸 𝘥𝘦𝘧𝘪𝘯𝘪𝘵𝘪𝘰𝘯𝘴:

𝘼) An Orchestrator in the GenAI system application is the central piece of software that orchestrates the end-to-end process. Think of apps using LangChain, LlamaIndex or Haystack.
𝘽) Trace is the end-to-end application flow from the entry point till the answer is produced, it is composed of smaller pieces called spans.
𝘾) Span is a smaller piece of the application flow that represents an atomic action like a function call or a database query. They can be sequential, or run in parallel.

ℹ️ As part of span we capture general metadata like start and end time, inputs and outputs of the span. On top of this metadata we track information specific to the GenAI system elements.

What might a trace look like for a naive RAG system?

𝟭. A query that has been submitted to the chat application.
𝟮. The query is embedded into a vector. 

✅ Additional metadata like input token count is persisted with the span so that we can estimate the cost of the procedure.

𝟯. ANN lookup performed against the Vector DB to retrieve the most relevant context. 

✅ Additional metadata about the query is persisted as part of the span together with the retrieved pieces of context and their relevance.

𝟰. A prompt is constructed from the system prompt and retrieved context.
𝟱. The prompt is passed to the LLM to construct the answer. 

✅ Additional metadata about input and output token count is captured together with the span so that we can estimate the cost of the procedure.

𝘞𝘩𝘺 𝘪𝘴 𝘵𝘳𝘢𝘤𝘪𝘯𝘨 𝘰𝘧 𝘎𝘦𝘯𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘪𝘮𝘱𝘰𝘳𝘵𝘢𝘯𝘵?

- These applications are usually complex chains, errors can happen in different steps of your application. E.g. Embedding of query is taking longer than expected or you have reached API limits of LLM provider.
- Cost for calling LLM APIs will be variable depending on the length of inputs and produced outputs. You would usually trace this information and analyze it to help forecast expenses.
- GenAI systems are non-deterministic and will deteriorate over time. They need to be evaluated on span level rather than input/output of the entire system so that you can tune each piece separately.
- …

Are you tracing your Agents? Let me know in the comments 👇

𝗧𝗼𝗼𝗹 𝗖𝗮𝗹𝗹𝗶𝗻𝗴 𝘃𝗶𝗮 𝗠𝗖𝗣 vs. implementing it 𝗡𝗮𝘁𝗶𝘃𝗲 𝘁𝗼 𝘁𝗵𝗲 𝗔𝗴𝗲𝗻𝘁. What's the difference?

Recently there have been many MCP vs. Function Calling articles floating around. Let’s simplify. 

❗️ 𝗖𝗹𝗮𝗿𝗶𝗳𝘆𝗶𝗻𝗴 𝗼𝗻𝗲 𝗱𝗲𝘁𝗮𝗶𝗹: Function Calling and Tool Use in Agentic Systems are almost the same thing. You can implement tools via functions, the only difference is that functions are usually used to enforce stricter structure to the input and output schema.

𝘜𝘴𝘪𝘯𝘨 𝘔𝘊𝘗 𝘵𝘰 𝘦𝘹𝘱𝘰𝘴𝘦 𝘵𝘰𝘰𝘭𝘴 𝘵𝘰 𝘺𝘰𝘶𝘳 𝘈𝘨𝘦𝘯𝘵𝘴:

ℹ️ In this case your Agent becomes an MCP Host and implements one or more MCP Clients to communicate with MCP Servers.

𝟭. User Query is passed to the Agent (usually a Python application).
𝟮. The application implements MCP Client and via it retrieves all available tools from the MCP servers.
𝟯. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which tools need to be invoked and with what parameters.
𝟰. The Agent application communicates with the MCP server (via MCP Client) again and sends the tool execution request. After execution completes the Agent receives the required data.
𝟱. User Query is sent to the LLM together with the data retrieved by the Tool calls.
𝟲. The answer is constructed and returned to the user via the Agent.

𝘜𝘴𝘪𝘯𝘨 𝘕𝘢𝘵𝘪𝘷𝘦 𝘍𝘶𝘯𝘤𝘵𝘪𝘰𝘯 𝘊𝘢𝘭𝘭𝘪𝘯𝘨:

𝟭. User Query is passed to the Agent (usually a Python application).
𝟮. All of the available Functions/Tools are defined as part of the Agent code (procedural memory).
𝟯. The list of available Tools is passed together with the User Query to a LLM via a prompt. The LLM figures out which functions need to be invoked and with what parameters.
𝟰. The Agent application directly executes the functions.
𝟱. User Query is sent to the LLM together with the data retrieved after function execution.
𝟲. The answer is constructed and returned to the user via the Agent.

𝗠𝘆 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ Will MCP eat up the practice of Native Function Calling? If we adopt the standard then I believe so.
❗️ Will MCP compete with LLM Orchestration frameworks? I believe MCP will replace the Tool use abstractions as they are used today and frameworks will be responsible mostly for managing the topology and state of the Agentic systems long term.

Are you using MCP already to expose Tools to your Agents? Let me know in the comments. 👇

Some 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 might seem simple from the outside. They are not 👇

Here are some of the layers that are hidden from you as a user of agentic

We usually start building and experimenting with Raw Model APIs. These rely on complex underlying infrastructure.

𝟭. 𝘎𝘗𝘜/𝘊𝘗𝘜 Resources.
𝟮. 𝘉𝘢𝘴𝘦 𝘐𝘯𝘧𝘳𝘢𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦 that orchestrates Model deployment. Think Kubernetes, Slurm, vLLM.
𝟯. 𝘍𝘰𝘶𝘯𝘥𝘢𝘵𝘪𝘰𝘯 𝘔𝘰𝘥𝘦𝘭𝘴 themselves that required tens of millions of dollars to be trained.

To produce a reliable MVP you will need some internal data and visibility to your system. This will require you to make choices in how you:

𝟰. 𝘚𝘵𝘰𝘳𝘦 the data: Vector DBs, Graph DBs etc.
𝟱. 𝘖𝘣𝘴𝘦𝘳𝘷𝘦 𝘓𝘓𝘔 interactions to see into the actions your system is performing. You will need this for debugging and evolving your Agents.

Scaling up requires more stability and insights into what is happening inside of the system.

𝟲. 𝘌𝘷𝘢𝘭𝘶𝘢𝘵𝘪𝘰𝘯 helps you to go beyond passively observing the system to proactively monitoring it by bringing automation via evaluation rules that are applied against steps performed by your Agents.
𝟳. 𝘖𝘳𝘤𝘩𝘦𝘴𝘵𝘳𝘢𝘵𝘪𝘰𝘯 of the system: LLM Orchestration frameworks help with solving issues like retries, chaining your prompts, tool calling etc.

To expose your application to the general public you would need additional automations and guardrails to prevent disasters that could shut your business in seconds.

𝟴. 𝘔𝘰𝘥𝘦𝘭 𝘙𝘰𝘶𝘵𝘪𝘯𝘨 helps in choosing best LLMs for your prompts, prompt management, fallback mechanisms in case of unresponsive LLM APIs or hitting API limits. Etc.
𝟵. 𝘚𝘦𝘤𝘶𝘳𝘪𝘵𝘺 is critical to avoid disasters related to data leakage. Think Guardrails, Red Teaming etc.

❗️And these are just basic requirements, there is more 🙂

Come in the emerging requirements:

𝟭𝟬. Memory.
𝟭𝟭. Agent Communication Protocols.
...

Anything I am missing? Let me know in the comments 👇

What does an 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 look like? 
 
MLOps practices are there to improve Machine Learning Product development velocity, the biggest bottlenecks happen when Experimentation Environments and other infrastructure elements are integrated poorly. 
 
Let’s look into the properties that an effective Experimentation Environment should have. As a MLOps engineer you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for.
 
𝟭: Access to the raw data. While handling raw data is the responsibility of Data Engineering function, Data Scientists need the ability to explore and analyze available raw data and decide which of it needs to be moved upstream the Data Value Chain (2.1). 
 
𝟮: Access to the curated data. Curated data might be available in the Data Warehouse but not exposed via a Feature Store. Such Data should not be exposed for model training in production environments. Data Scientists need the ability to explore curated data and see what needs to be pushed downstream (3.1). 
 
𝟯: Data used for training of Machine Learning models should be sourced from a Feature Store if the ML Training pipeline is ready to be moved to the production stage. 
 
𝟰: Data Scientists should be able to easily spin up different types of compute clusters - might it be Spark, Dask or any other technology - to allow effective Raw and Curated Data exploration. 
 
𝟱: Data Scientists should be able to spin up a production like remote Machine Learning Training pipeline in development environment ad-hoc from the Notebook, this increases speed of iteration significantly. 
 
𝟲: There should be an automated setup in place that would perform the testing and promotion to a higher env when a specific set of Pull Requests are created. E.g. a PR from feature/* to release/* branch could trigger a CI/CD process to test and deploy the ML Pipeline to a pre-prod environment. 
 
𝟳: Notebooks and any additional boilerplate code for CI/CD should be part of your Git integration. Make it crystal clear where a certain type of code should live - a popular way to do this is providing repository templates with clear documentation. 
 
𝟴: Experiment/Model Tracking System should be exposed to both local and remote pipelines. 
 
𝟗: Notebooks have to be running in the same environment that your production code will run in. Incompatible dependencies should not cause problems when porting applications to production. It can be achieved by running Notebooks in containers. 
 
Did I miss something? Let me know your thoughts! 👇

#AI #MachineLearning

Fusion of 𝗥𝗔𝗚 (Retrieval Augmented Generation) and 𝗖𝗔𝗚 (Cache Augmented Generation). How can you benefit from it as AI Engineer?

Few months ago there was a lot of hype around a technique called CAG. While it is powerful to its own extent, the real magic happens when you combine CAG with regular RAG. Let’s see what it would look like and what additional considerations should be taken into account.

Here are example steps to implement CAG + RAG architecture:

𝘋𝘢𝘵𝘢 𝘗𝘳𝘦𝘱𝘳𝘰𝘤𝘦𝘴𝘴𝘪𝘯𝘨:

𝟭. We use only rarely changing data sources for Cache Augmented Generation. On top of the requirement of data changing rarely we should also think about which of the sources are often hit by relevant queries. Once we have this information, only then we pre-compute all of this selected data into a KV Cache of the LLM. Cache it in memory. This only needs to be done once, the following steps can be run multiple times without recomputing the initial cache.
𝟮. For RAG, if necessary, precompute and store vector embeddings in a compatible database to be searched later in step 4. Sometimes simpler data types are enough for RAG, a regular database might suffice.

𝘘𝘶𝘦𝘳𝘺 𝘗𝘢𝘵𝘩:

We can now utilise the preprocessed data.

𝟯. Compose a prompt including user query and the system prompt with instructions on how cached context and retrieved external context should be used by the LLM.
𝟰. Embed a user query to be used for semantic search via vector DBs and query the context store to retrieve relevant data. If semantic search is not required, query other sources, like real time databases or web.
𝟱. Enrich the final prompt with external context retrieved in step 4.
𝟲. Return the final answer to the user.

𝘚𝘰𝘮𝘦 𝘊𝘰𝘯𝘴𝘪𝘥𝘦𝘳𝘢𝘵𝘪𝘰𝘯𝘴:

➡️ Context window is not infinite and even while some models boast enormous context window sizes, the needle in the haystack problem has not yet been solved so use available context wisely and cache only the data you really need.

✅ For some business cases, specific datasets are extremely valuable to be passed to the model as cache. Think about an assistant that has to always comply with a lengthy set of internal rules stored in multiple documents. 
✅ While CAG has been popularised for Open Source just recently, it is already viable for some time via Prompt Caching features in OpenAI and Anthropic APIs. It is really easy to start prototyping there.
✅ You should always separate hot and cold data sources, only use cold (data that changes rarely) in your cache, otherwise the data will go stale and the application will go out of sync.

❌ Be very careful about what you cache as the data will be available for all users to query.
❌ It is very hard to ensure RBAC for cached data unless you have a separate model with its own cache per role.

Have you used the fusion of CAG and RAG already? Let me know about your results in the comments 👇

Siddharth Swain

Building even a simple 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗴𝗿𝗮𝗱𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) 𝗯𝗮𝘀𝗲𝗱 𝗔𝗜 𝘀𝘆𝘀𝘁𝗲𝗺 is a challenging task. Read until the end to understand why 👇

Here are some of the moving parts in the RAG based systems that you will need to take care of and continuously tune in order to achieve desired results:

𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹:

𝘍 ) Chunking - how do you chunk the data that you will use for external context.

- Small, Large chunks.
- Sliding or tumbling window for chunking.
- Retrieve parent or linked chunks when searching or just use originally retrieved data.

𝘊 ) Choosing the embedding model to embed and query and external context to/from the latent space. Considering Contextual embeddings.

𝘋 ) Vector Database.

- Which Database to choose.
- Where to host.
- What metadata to store together with embeddings.
- Indexing strategy.

𝘌 ) Vector Search

- Choice of similarity measure.
- Choosing the query path - metadata first vs. ANN first.
- Hybrid search.

𝘎 ) Heuristics - business rules applied to your retrieval procedure.

- Time importance.
- Reranking.
- Duplicate context (diversity ranking).
- Source retrieval.
- Conditional document preprocessing.

𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻:

𝘈 ) LLM - Choosing the right Large Language Model to power your application.

✅ It is becoming less of a headache the further we are into the LLM craze. The performance of available LLMs are converging, both open source and proprietary. The main choice nowadays is around using a proprietary model or self-hosting.

𝘉 ) Prompt Engineering - having context available for usage in your prompts does not free you from the hard work of engineering the prompts. You will still need to align the system to produce outputs that you desire and prevent jailbreak scenarios.

And let’s not forget the less popular part:

𝘏) Observing, Evaluating, Monitoring and Securing your application in production!

What other pieces of the system am I missing? Let me know in the comments 👇

#AI #LLM #MachineLearning

taplio

Aurimas Griciūnas's Linkedin Analytics