Matthew Blasa

Bridging the gap between data scientists and the business.

Learn more: 
AI Product and Strategy Substack:  https://datalife360.substack.com/
Pragmatic Data Scientist: https://pragmaticdatascientist.substack.com/
Linkedin Learning: https://shorturl.at/hg0Bc

Strategy Consultant

Interested in a discovery session?
email: info@aspire360.ai or DM me for more information.

Aurimas Griciūnas

Senior Solutions Architect

Follow me to learn about Data Engineering and ML Systems | Author of SwirlAI Newsletter | Public Speaker

Tired of all the noise surrounding AI space in recent years? Having a hard time keeping up with all of the news in the field and distinguishing what is really important? Follow me and subscribe to my newsletter to get actionable insights on building production ready AI Systems and keep track of only the important signal of what is happening in the AI world.

With over a decade of experience in Data Science, AI/ML Engineering, and MLOps, I’m also dedicated to helping practitioners level up their skills in MLOps, Data Engineering, Machine Learning, and related fields. I share my knowledge in a clear and accessible way to make learning straightforward and actionable.

If you want to partner with SwirlAI and get your product in front of 40K+ AI/ML/Data Engineers and Leaders via our Newsletter or 135K+ AI/ML/Data Engineers and Leaders via LinkedIn, contact me directly.

Head of Product

Leading Product Strategy.

Fusion of 𝗥𝗔𝗚 (Retrieval Augmented Generation) and 𝗖𝗔𝗚 (Cache Augmented Generation). How can you benefit from it as AI Engineer?

Let’s see what it looks like and what additional considerations should be taken into account.

Here are example steps to implement CAG + RAG architecture:

𝘋𝘢𝘵𝘢 𝘗𝘳𝘦𝘱𝘳𝘰𝘤𝘦𝘴𝘴𝘪𝘯𝘨:

𝟭. We use only rarely changing data sources for Cache Augmented Generation. On top of the requirement of data changing rarely we should also think about which of the sources are often hit by relevant queries. Once we have this information, only then we pre-compute all of this selected data into a KV Cache of the LLM. Cache it in memory. This only needs to be done once, the following steps can be run multiple times without recomputing the initial cache.
𝟮. For RAG, if necessary, precompute and store vector embeddings in a compatible database to be searched later in step 4. Sometimes simpler data types are enough for RAG, a regular database might suffice.

𝘘𝘶𝘦𝘳𝘺 𝘗𝘢𝘵𝘩:

We can now utilise the preprocessed data.

𝟯. Compose a prompt including user query and the system prompt with instructions on how cached context and retrieved external context should be used by the LLM.
𝟰. Embed a user query to be used for semantic search via vector DBs and query the context store to retrieve relevant data. If semantic search is not required, query other sources, like real time databases or web.
𝟱. Enrich the final prompt with external context retrieved in step 4.
𝟲. Return the final answer to the user.

𝘚𝘰𝘮𝘦 𝘊𝘰𝘯𝘴𝘪𝘥𝘦𝘳𝘢𝘵𝘪𝘰𝘯𝘴:

➡️ Context window is not infinite and even while some models boast enormous context window sizes, the needle in the haystack problem has not yet been solved so use available context wisely and cache only the data you really need.

✅ For some business cases, specific datasets are extremely valuable to be passed to the model as cache. Think about an assistant that has to always comply with a lengthy set of internal rules stored in multiple documents. 
✅ While CAG has been popularised for Open Source just recently, it is already viable for some time via Prompt Caching features in OpenAI and Anthropic APIs. It is really easy to start prototyping there.
✅ You should always separate hot and cold data sources, only use cold (data that changes rarely) in your cache, otherwise the data will go stale and the application will go out of sync.

❌ Be very careful about what you cache as the data will be available for all users to query.
❌ It is very hard to ensure RBAC for cached data unless you have a separate model with its own cache per role.

Learn how to apply the technique in my End-to-end AI Engineering Bootcamp. Next cohort kicking off on January 12th: https://lnkd.in/dagWE5r3

🎁 Code LASTCHANCE for 10% off.

Founder @ SwirlAI • Ex-CPO @ neptune.ai (Acquired by OpenAI) • UpSkilling the Next Generation of AI Talent • Author of SwirlAI Newsletter • Public Speaker

𝗠𝗖𝗣 plus 𝗔𝟮𝗔, here is how they complement each other 👇

Protocol wars continue to rage, let's understand how Googles A2A (Agent2Agent) protocol is different from MCP and how they complement each other (read till the end).

𝘔𝘰𝘷𝘪𝘯𝘨 𝘱𝘪𝘦𝘤𝘦𝘴 𝘪𝘯 𝘔𝘊𝘗:

𝟭. MCP Host - Programs using LLMs at the core that want to access data through MCP.

❗️ When combined with A2A, an Agent becomes MCP Host.

𝟮. MCP Client - Clients that maintain 1:1 connections with servers.
𝟯. MCP Server - Lightweight programs that each expose specific capabilities through the standardised Model Context Protocol.
𝟰. Local Data Sources - Your computer’s files, databases, and services that MCP servers can securely access.
𝟱. Remote Data Sources - External systems available over the internet (e.g., through APIs) that MCP servers can connect to.

𝘌𝘯𝘵𝘦𝘳 𝘈2𝘈:

Where MCP falls short, A2A tries to help. In multi-Agent applications where state is not necessarily shared

𝟲. Agents (MCP Hosts) would implement and communicate via A2A protocol, that enables:

➡️ Secure Collaboration.
➡️ Task and State Management.
➡️ User Experience Negotiation.
➡️ Capability discovery - similar to MCP tools.

𝗛𝗼𝗻𝗲𝘀𝘁 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝘀:

❗️ Open protocols for Agent communication are important but they are just a piece of the picture, there will be a need for standards that govern all of the existing protocols and other missing pieces. E.g.

❓ How do we standardise tracing and Observability in multi-agent IOA systems?
❓ How do we retain the identity of the running job of an AI Agent instance if the communication standard is not unified in different parts of the pipeline?
❓ ... 

✅ More on this in my future posts, stay tuned!

Let me know your thoughts in the comments. 👇

A breakdown of 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗶𝗻 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 👇 And yes, it can also be used for LLM based systems!

It is critical to ensure Data Quality and Integrity upstream of ML Training and Inference Pipelines, trying to do that in the downstream systems will cause unavoidable failure when working at scale.

There is a ton of work to be done on the Data Lake or LakeHouse layer. 𝗦𝗲𝗲 𝘁𝗵𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗯𝗲𝗹𝗼𝘄.

𝘌𝘹𝘢𝘮𝘱𝘭𝘦 𝘢𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵𝘶𝘳𝘦 𝘧𝘰𝘳 𝘢 𝘱𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯 𝘨𝘳𝘢𝘥𝘦 𝘦𝘯𝘥-𝘵𝘰-𝘦𝘯𝘥 𝘥𝘢𝘵𝘢 𝘧𝘭𝘰𝘸:

𝟭: Schema changes are implemented in version control, once approved - they are pushed to the Applications generating the Data, Databases holding the Data and a central Data Contract Registry.

Applications push generated Data to Kafka Topics:

𝟮: Events emitted directly by the Application Services.

👉 This also includes IoT Fleets and Website Activity Tracking.

𝟮.𝟭: Raw Data Topics for CDC streams.

𝟯: A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Contract Registry.
𝟰: Data that does not meet the contract is pushed to Dead Letter Topic.
𝟱: Data that meets the contract is pushed to Validated Data Topic.
𝟲: Data from the Validated Data Topic is pushed to object storage for additional Validation.
𝟳: On a schedule Data in the Object Storage is validated against additional SLAs in Data Contracts and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes.
𝟴: Modeled and Curated data is pushed to the Feature Store System for further Feature Engineering.
𝟴.𝟭: Real Time Features are ingested into the Feature Store directly from Validated Data Topic (5).

👉 Ensuring Data Quality here is complicated since checks against SLAs is hard to perform.

𝟵: High Quality Data is used in Machine Learning Training Pipelines.
𝟭𝟬: The same Data is used for Feature Serving in Inference.

Note: ML Systems are plagued by other Data related issues like Data and Concept Drifts. These are silent failures and while they can be monitored, we can’t include it in the Data Contract.

Let me know your thoughts! 👇

The main 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝗧𝘆𝗽𝗲𝘀 👇

It’s no secret that an AI Agent is a piece of software wrapping an LLM or multiple LLMs as a reasoning engine to guide its execution flow. The deployment types for these agents also closely resemble those of software or ML applications, here are the main ones:

➡️ 𝗕𝗮𝘁𝗰𝗵:

- You run your Agents as a part of ETL/ELT Process on a given schedule.
- An agent takes the form of a CLI script most of the time and is scheduled to run periodically via tools like Apache Airflow.
- You consume the accumulated work, connect to external context and tools if needed and save the agent's results to a batch storage.
- Computation results can be loaded into a real-time storage and used for real time applications as well.

➡️ 𝗘𝗺𝗯𝗲𝗱𝗱𝗲𝗱 𝗶𝗻 𝗮 𝗦𝘁𝗿𝗲𝗮𝗺 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻:

- You apply your agentic logic as a part of a Stream Processing Pipeline.
- While work events are continuously piped through your Streaming Data Pipelines, an agent application applies computation on this data and returns it to the system - most likely another Streaming Storage.
- This deployment type can also easily connect to any required external context storage or tools.
- Outputs can be consumed by multiple applications subscribing to the Inference Stream.

✅ This is an ideal deployment type for Ambient Agents, they can listen for events flowing through the system and pick up the ones dedicated to them.

➡️ 𝗥𝗲𝗮𝗹 𝗧𝗶𝗺𝗲:

- You expose your agent as a Backend Service (REST or gRPC).
- This agent retrieves any required external context from external context stores as well as utilising tools available to it.
- Computation can be requested by any application in real time.

➡️ 𝗘𝗱𝗴𝗲:

- You embed your agent's logic directly into the application that runs on a user device.
- This method provides the lowest latency and improves privacy.
- Data in most cases is generated and lives inside of the device significantly improving the security.

Learn how to apply these in practice in my End-to-end AI Engineering Bootcamp: https://lnkd.in/dagWE5r3

Kicking off next Monday, use code LASTCHANCE for 10% discount!

What types of deployments are you mostly working on? Let me know in the comments! 👇

Founder @ SwirlAI • UpSkilling the Next Generation of AI Talent • Author of SwirlAI Newsletter • Public Speaker

𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 is a must have in your tool belt as an AI Engineer. 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 sits at the core of it, why is it important?

Tracing and instrumentation of software have been around for decades now. With AI systems resembling regular software even more, we are now moving the practice here as well (with a few key differences).

Let’s look into the process of tracing from a perspective of a naive RAG system.

𝘍𝘦𝘸 𝘥𝘦𝘧𝘪𝘯𝘪𝘵𝘪𝘰𝘯𝘴:

𝘼) An Orchestrator in the GenAI system application is the central piece of software that orchestrates the end-to-end process. Think of apps using LangChain, LlamaIndex or Haystack.
𝘽) Trace is the end-to-end application flow from the entry point till the answer is produced, it is composed of smaller pieces called spans.
𝘾) Span is a smaller piece of the application flow that represents an atomic action like a function call or a database query. They can be sequential, or run in parallel.

ℹ️ As part of span we capture general metadata like start and end time, inputs and outputs of the span. On top of this metadata we track information specific to the GenAI system elements.

What might a trace look like for a naive RAG system?

𝟭. A query that has been submitted to the chat application.
𝟮. The query is embedded into a vector. 

✅ Additional metadata like input token count is persisted with the span so that we can estimate the cost of the procedure.

𝟯. ANN lookup performed against the Vector DB to retrieve the most relevant context. 

✅ Additional metadata about the query is persisted as part of the span together with the retrieved pieces of context and their relevance.

𝟰. A prompt is constructed from the system prompt and retrieved context.
𝟱. The prompt is passed to the LLM to construct the answer. 

✅ Additional metadata about input and output token count is captured together with the span so that we can estimate the cost of the procedure.

𝘞𝘩𝘺 𝘪𝘴 𝘵𝘳𝘢𝘤𝘪𝘯𝘨 𝘰𝘧 𝘎𝘦𝘯𝘈𝘐 𝘴𝘺𝘴𝘵𝘦𝘮𝘴 𝘪𝘮𝘱𝘰𝘳𝘵𝘢𝘯𝘵?

- These applications are usually complex chains, errors can happen in different steps of your application. E.g. Embedding of query is taking longer than expected or you have reached API limits of LLM provider.
- Cost for calling LLM APIs will be variable depending on the length of inputs and produced outputs. You would usually trace this information and analyze it to help forecast expenses.
- GenAI systems are non-deterministic and will deteriorate over time. They need to be evaluated on span level rather than input/output of the entire system so that you can tune each piece separately.
- …

Are you tracing your Agents? Let me know in the comments 👇

Don't get fooled, building a 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗴𝗿𝗮𝗱𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) 𝗯𝗮𝘀𝗲𝗱 𝗔𝗜 𝘀𝘆𝘀𝘁𝗲𝗺 is a challenging task. Read until the end to understand why 👇

Here are some of the moving parts in the RAG based systems that you will need to take care of and continuously tune in order to achieve desired results:

𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹:

𝘍 ) Chunking - how do you chunk the data that you will use for external context.

- Small, Large chunks.
- Sliding or tumbling window for chunking.
- Retrieve parent or linked chunks when searching or just use originally retrieved data.

𝘊 ) Choosing the embedding model to embed and query and external context to/from the latent space. Considering Contextual embeddings.

𝘋 ) Vector Database.

- Which Database to choose.
- Where to host.
- What metadata to store together with embeddings.
- Indexing strategy.

𝘌 ) Vector Search

- Choice of similarity measure.
- Choosing the query path - metadata first vs. ANN first.
- Hybrid search.

𝘎 ) Heuristics - business rules applied to your retrieval procedure.

- Time importance.
- Reranking.
- Duplicate context (diversity ranking).
- Source retrieval.
- Conditional document preprocessing.

𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻:

𝘈 ) LLM - Choosing the right Large Language Model to power your application.

✅ It is becoming less of a headache the further we are into the LLM craze. The performance of available LLMs are converging, both open source and proprietary. The main choice nowadays is around using a proprietary model or self-hosting.

𝘉 ) Prompt Engineering - having context available for usage in your prompts does not free you from the hard work of engineering the prompts. You will still need to align the system to produce outputs that you desire and prevent jailbreak scenarios.

And let’s not forget the less popular part:

𝘏) Observing, Evaluating, Monitoring and Securing your application in production!

What other pieces of the system am I missing? Let me know in the comments 👇

If you want to start learning Kubernetes as an AI Engineer, look no further 👇

Some time ago I created a 2 part series on the main concepts you should know as a developer working with K8s. Examples and walkthroughs included - everything runs locally!

In the two articles you will learn:

➡️ Why Kubernetes for AI and Data Engineering.
➡️ General Kubernetes Resources for application configuration.
➡️ How to deploy applications.
➡️ Different ways to deploy applications and when to use which.
➡️ Visual explanation for all of the above.

Since I have resumed the newsletter, there will be many hands-on AI projects utilising Kubernetes for deployment. Be sure to sharpen your skills if you intend to follow! This resource is perfect for that.

Link to the first article: https://lnkd.in/ed2HfHjV

It is completely free 😉

taplio

The Battle

Matthew Blasa

Aurimas Griciūnas

Followers

Posts

Engagements

Likes

Performance ScoreAurimas Griciūnas's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Matthew Blasa

Aurimas Griciūnas

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Aurimas Griciūnas's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreAurimas Griciūnas's LinkedIn Performance