Paul Iusztin's Linkedin Analytics

Fine-tuning isn’t hard. Here's where most pipelines fall apart: Integrating it into a full LLM system. So here’s how we architected our training pipeline: 𝗜𝗻𝗽𝘂𝘁𝘀 𝗮𝗻𝗱 𝗼𝘂𝘁𝗽𝘂𝘁𝘀 The training pipeline has one job: → Input: • A dataset from the data registry • A base model from the model registry → Output: • A fine-tuned model registered in a model registry and ready for deployment In our case: - Base: Llama 3.1 8B Instruct • Dataset: Custom summarization data generated from web documents • Output: A specialized model that summarizes web content 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝘀𝘁𝗲𝗽𝘀 1. Load base model → apply LoRA adapters 2. Load dataset → format using Alpaca-style instructions 3. Fine-tune with Unsloth AI on T4 GPUs (via Colab) 4. Track training + eval metrics with Comet 5. If performance is good → push to Hugging Face model registry 6. If not → iterate with new data or hyperparameters Most research happens in Notebooks (and that’s okay). So we kept our training pipeline in a Jupyter Notebook on Colab. Why? → Let researchers feel at home → No SSH friction → Visualize results fast → Enable rapid iteration → Plug into the rest of the system via registries Just because it’s manual doesn’t mean it’s isolated. Here's how it connects: - Data registry: feeds in the right fine-tuning set • Model registry: stores the fine-tuned weights • Inference service: serves the fine-tuned model solely using the model registry • Eval tracker: logs metrics + compares runs in real-time The Notebook is completely decoupled from the rest of the LLM system. Can it be automated? Yes... and we’re almost there. With ZenML already managing our offline pipelines, the training code can be converted to a deployable pipeline. The only barrier? Cost and compute. That’s why continuous training (CT) in the LLM space is more of a dream than something that you actually want to do in practice. TL;DR: If you’re thinking of training your own LLMs, don’t just ask “how do I fine-tune this?” Ask: • How does it integrate? • What data version did I use? • Where do I store the weights? • How do I track experiments across runs? • How can I detach the fine-tuning from deployment? That’s what separates model builders from AI engineers. 🔗 Full breakdown here: https://lnkd.in/de_ndNbQ

Fine-tuning should NEVER be the first step when building an AI system. Here’s the only time you should do it: When nothing else works. But let's face it... Most teams jump straight into fine-tuning Why? Because it feels technical. Custom. Smart. In reality, it’s often just unnecessary complexity. Before you spend hours generating synthetic data and burning through GPUs, you must ask yourself three questions: → Can I solve this with smart prompt engineering? → Can I improve it further by adding RAG? → Have I even built an evaluatable system yet? If the answer to those isn’t a solid "YES," you have no business fine-tuning anything. I say this all the time - "You don’t need your own model; you need better system design." - Prompt engineering handles ~30–50% of cases - RAG handles another ~30–40% - Fine-tuning? Reserve it for the last 10% (when the problem demands it) For example, in our work at Decoding ML, we only fine-tune when: - The context window is too small for RAG to help - The task requires domain-specific tone, behavior, or reasoning - The system is mature enough to warrant the extra complexity Anything sooner is overkill. Thanks to Maxime Labonne for helping sharpen this thinking during our work on The LLM Engineer’s Handbook (especially when mapping tradeoffs between fine-tuning, prompting, and RAG) Want to learn more? Check out Lesson 4 of the Second Brain AI Assistant course. Link in the comments.

90% of RAG systems struggle with the same bottleneck: (And better LLMs are not the solution) It's retrieval. And most teams don’t realize it because they rush to build without proper evaluation. Before I tell you how to fix this, let me make something clear - 𝗡𝗮𝗶𝘃𝗲 𝗥𝗔𝗚 𝗶𝘀 𝗲𝗮𝘀𝘆. You chunk some docs, embed them, drop a top_k retriever on top, and call it a pipeline. Getting it production-ready? That’s where most teams stall. → They get hallucinations. → They miss key info. → Their outputs feel... off. Why? Because the quality of generation is downstream of the quality of context. ... and naive RAG often pulls in irrelevant or partial chunks that confuse the LLM. If you're serious about improving your system, here's the progression that actually works: 𝗦𝘁𝗲𝗽 𝟭: 𝗙𝗶𝘅 𝘁𝗵𝗲 𝗕𝗮𝘀𝗶𝗰𝘀 These “table-stakes” upgrades outperform fancy models most of the time: → Smarter Chunking - Dynamic over fixed-size. Respect structure. → Chunk Size Tuning - Too long = loss in the middle. Too short = fragmented context. → Metadata Filtering - Boosts precision by narrowing scope semantically and structurally. → Hybrid Search - Combine vector + keyword filtering. 𝗦𝘁𝗲𝗽 𝟮: 𝗟𝗮𝘆𝗲𝗿 𝗼𝗻 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 When basic techniques aren’t enough: → Re-ranking (learned or rule-based) → Small-to-Big Retrieval: Retrieve sentences, synthesize larger windows. → Recursive Retrieval (e.g., LlamaIndex) → Multi-hop + agentic retrieval: When you need reasoning across documents. 𝗦𝘁𝗲𝗽 𝟯: 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝗼𝗿 𝗗𝗶𝗲 𝗧𝗿𝘆𝗶𝗻𝗴 There's no point iterating blindly. Do the following: → End-to-End eval - Is the output good? Ground truths, synthetic evals, user feedback. → Component-level eval - Does the retriever return the right chunks? Use ranking metrics like MRR, NDCG, success@k. 𝗦𝘁𝗲𝗽 𝟰: 𝗙𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 = 𝗟𝗮𝘀𝘁 𝗥𝗲𝘀𝗼𝗿𝘁 Don’t start here. Do this only when: → Your domain is so specific general embeddings fail. → Your LLM is too weak to synthesize even when context is correct. → You’ve squeezed all juice from prompt + retrieval optimizations. Fine-tuning adds cost, latency, and infra complexity. It’s powerful, but only when everything else is dialed in. 𝗡𝗼𝘁𝗲: These notes are from a talk over a year old. And yet... most teams are still stuck in Step 0. That tells you something - The surface area of RAG is small. But building good RAG is still an unsolved craft. Let’s change that. Want to learn to implement advanced RAG systems yourself? The link is in the comments. 𝗜𝗺𝗮𝗴𝗲 𝗰𝗿𝗲𝗱𝗶𝘁: LlamaIndex and Jerry Liu

Everyone’s building agents. But very few are building them for production... And that’s the gap we wanted to close with Lesson 2 of the 𝗣𝗵𝗶𝗹𝗼𝗔𝗴𝗲𝗻𝘁𝘀 course. Too often, agentic demos look impressive - until you try scaling them. Then comes the hidden complexity: → Orchestrating LLM calls → Managing memory → Debugging emergent behavior → Building in retrieval without breaking the flow That’s why this lesson doesn’t stop at “toy demos.” We show you how to build a real, production-ready RAG agent inside a gaming simulation. An agent that can impersonate philosophers, carry context-aware conversations, and dynamically adapt to user input. Not just an NPC, but a character. Here’s what you’ll build: → An agentic RAG system powered by LangGraph → A memory architecture backed by MongoDB → Persona-specific prompt templates streamed via Groq LLM APIs → Observability and evaluation pipelines instrumented with Opik (by Comet) → A system designed to scale, recover, and impersonate in real time This is how you go from scripts to systems. From chatbots to characters. Lesson 2 is live. (Link in the comments) P.S. A massive shout out and thanks to Miguel Otero Pedrido for the collab 👊

Finding the right open-source LLMs to work with is a pain in the backside. 98% of LLM leaderboards are bloated. Too many closed models. Too many broken repos. Too little clarity on what actually works in production. It's frustrating. Fortunately, I found something to help with mitigate this issue... If you’re looking for open-source LLMs that just run - For fine-tuning, quantization, and deployment... Unsloth AI has done the hard work for you. They’ve compiled a list of all the popular, supported, and production-viable models that: → Fine-tune easily (with Unsloth + QLoRA) → Quantize to GGUFs for local inference (Ollama, llama.cpp, OpenWebUI) → Play well with Hugging Face and Python → Come with working code and notebook examples → Easy to deploy to Hugging Face Inference Endpoints, AWS, GCP, Modal, and more No more jumping between broken GitHub repos or guessing which models will survive a production pipeline. It’s the fastest way to stay current without losing your mind. If you’re working with open-source LLMs, just bookmark this list. Link in the comments!

90% of AI engineers are dangerously abstracted from reality. They work with: → Prebuilt models → High-level APIs → Auto-magical cloud tools But here’s the thing - If you don’t understand how these tools actually work, you’ll always be guessing when something breaks. That’s why the best AI engineers I know go deeper... They understand: How Git actually tracks changes. How Redis handles memory. How Docker isolates environments. If you’re serious about engineering, you'd go build the tools you use. And it’s why I recommend CodeCrafters.io (YC S22) You won’t just learn tools. You’ll rebuild them (from scratch). → Git, Redis, Docker, Kafka, SQLite, Shell... → Step by step, test by test → In your favorite language (Rust, Python, Go, etc.) It’s perfect for AI engineers who want to: → Level up their backend + system design skills → Reduce debugging time in production → Build apps that actually scale under load And most importantly... → Stop being a model user → Start being a systems thinker If I had to level up my engineering foundations today, CodeCrafters is where I’d start. The ink is in the comments. P.S. We only promote tools we use or would personally take. P.S.S. Subscribe with my affiliate link to get a 40% discount :)

The difference between RAG and Agentic RAG isn’t technical. It’s philosophical... RAG assumes answers are linear. Agentic RAG assumes thinking is iterative. That single belief changes how you architect everything. Let me explain. Most RAG pipelines follow this recipe: → Embed a bunch of documents → Retrieve top-K chunks → Slam them into a prompt → Pray the model gets it right It works until the query gets complex. Then the whole thing falls apart. Why? Because RAG is passive. It retrieves once and hopes for the best. But real questions aren't solved with one shot. They evolve. They require clarification, follow-ups, and refined context. That’s where Agentic RAG comes in... Agentic RAG doesn’t just retrieve, it also reasons: → Do I have enough context? → Should I re-query with a better search? → Should I ask the user for clarification? → Which tool should I use next? The result? A system that thinks before it speaks. If you’re building copilots, assistants, or longform Q&A tools, this matters. Because reliability comes from better decisions. Agentic RAG introduces that decision loop. It turns workflows into systems. It trades static pipelines for dynamic reasoning. And that mindset shift is where real GenAI builders separate themselves from the hype. Want to see what Agentic RAG looks like in action? We break it down with code, graphs, and production use cases in the Second Brain AI Assistant course. 🔗 Link: https://lnkd.in/dA465E_J

The #1 mistake in building LLM agents? Thinking the project ends at reasoning. Here's when it actually ends: When your agent can talk to the world securely, reliably, and in real time. And that’s what 𝗟𝗲𝘀𝘀𝗼𝗻 𝟰 𝗼𝗳 𝘁𝗵𝗲 𝗣𝗵𝗶𝗹𝗼𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗼𝘂𝗿𝘀𝗲 is all about. Up to this point, we focused on making our agents think: → Philosophical worldviews → Context-aware reasoning → Memory-backed conversations But intelligence alone isn’t enough. To be useful, agents need a voice. To be deployable, they need an interface. To be real, they need to exist as APIs. This lesson is the bridge from the local prototype to the live system. Here’s what you’ll learn: → How to deploy your agent as a REST API using FastAPI → How to stream responses token-by-token with WebSockets → How to wire up a clean backend–frontend architecture using FastAPI (web server) + Phaser (game interface) → How to think about agent interfaces in real-world products (not just demos) In short: 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗵𝗼𝘄 𝘆𝗼𝘂 𝘀𝗵𝗶𝗽 𝗮𝗻 𝗮𝗴𝗲𝗻𝘁 𝘄𝗵𝗼 𝗿𝗲𝗮𝘀𝗼𝗻𝘀 𝗔𝗡𝗗 𝗿𝗲𝘀𝗽𝗼𝗻𝗱𝘀 𝗶𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻. Shoutout to Anca-Ioana Martin for helping shape this lesson and write the deep-dive article. And of course... big thanks to my co-creator Miguel Otero Pedrido for the ongoing collab. 🔗 Link to Lesson 4 in the comments.

Everyone chunks documents for retrieval. But what if that’s the wrong unit? Let me explain.. In standard RAG, we embed small text chunks and pass those into the LLM as context. It’s simple, but flawed. Why? Because small chunks are great for retrieval precision, but terrible for generation context. That’s where Parent Retrieval comes in. (aka small-to-big retrieval) Here’s how it works: → You split your documents into small chunks → You embed and retrieve using those small chunks → But you don’t pass the chunk to the LLM... → You pass the parent document that the chunk came from The result? → Precise semantic retrieval (thanks to small, clean embeddings that encode a single entity) → Rich generation context (because the LLM sees the broader section) → Fewer hallucinations → Less tuning needed around chunk size and top-k It’s one of the few advanced RAG techniques that work in production. No fancy agents. No latency bombs. No retraining. We break it all down (with diagrams and code examples) in 𝗟𝗲𝘀𝘀𝗼𝗻 𝟱 𝗼𝗳 𝘁𝗵𝗲 𝗦𝗲𝗰𝗼𝗻𝗱 𝗕𝗿𝗮𝗶𝗻 𝗔𝗜 𝗔𝘀𝘀𝗶𝘀𝘁𝗮𝗻𝘁 𝗰𝗼𝘂𝗿𝘀𝗲. 🔗 Link to the full lesson in the comments.

Here’s the problem with most AI books: They teach the model, not the system. Which is fine... until you try to deploy that model in production. That’s where everything breaks: - Your RAG pipeline is duct-taped together - Your eval framework is an afterthought - Your prompts aren’t versioned - Your architecture can’t scale That’s why Maxime and I wrote the LLM Engineer’s Handbook... We wanted to create a practical guide for AI engineers who build real world AI applications. This isn’t just another guide... It's a practical road map for designing and deploying real-world LLM systems. In the book, we cover: → Efficient fine-tuning workflows → RAG architectures → Evaluation pipelines with LLM-as-judge → Scaling strategies for serving + infra → MLOps + LLMOps patterns baked in Whether you’re building your first assistant or scaling your 10th RAG app... This book gives you the mental models and engineering scaffolding to do it right. 🔗 Here's the link to get your copy: https://lnkd.in/dVgFJtzF

Back in 2023, I was struggling to keep track of my notes. So I did something the black mirror producers would be proud of... I built a second brain. All I wanted was an AI-powered assistant connected to my knowledge base. Something I could use to recall notes, surface ideas, and help me think. But making it real wasn’t as simple as connecting a chatbot to Notion. To get it working, I had to build a full system: → A modular RAG pipeline to retrieve from custom notes → Ingest, crawl, and clean all my noisy resources regardless of their form → Real-time APIs to stream responses as I typed → A memory layer to track context across conversations → Observability and evaluation to measure what worked No hacks. No hardcoded prompts. Just an LLM agent that understood my notes and helped me reason through them. After building, I open-sourced the entire thing - code and lessons. And over the past year, thousands of engineers have cloned, forked, and built on top of it. This week, the GitHub repo 𝗽𝗮𝘀𝘀𝗲𝗱 𝟭,𝟬𝟬𝟬 𝘀𝘁𝗮𝗿𝘀. I just want to say a massive thank you to everyone who tried it, shared it, or built something new with it. And to those who haven’t seen it yet - 🔗 The link’s in the comments. P.S. Let me know what you’d create with it.

98% of people consume AI content. But only 2% are actually building with it (and we wanted to change that)... So we created 5 open-source project-based AI courses that teach you how to go from zero to production. Each course is built with developers in mind. Backed by best practices from MLOps, LLMOps, and modern software engineering. And 100% free. Here’s what’s inside: 𝗣𝗵𝗶𝗹𝗼𝗔𝗴𝗲𝗻𝘁𝘀 (𝘄𝗶𝘁𝗵 The Neural Maze) Build a character simulation engine that brings AI agents to life with memory, retrieval, and real-time dialogue powered by: - Groq - LangGraph (by LangChain) - Opik (by Comet) → Learn agents, RAG, persona design, and modular LLM architecture. 𝗦𝗲𝗰𝗼𝗻𝗱 𝗕𝗿𝗮𝗶𝗻 𝗔𝗜 𝗔𝘀𝘀𝗶𝘀𝘁𝗮𝗻𝘁 Build an AI assistant that chats with your personal knowledge base. → Learn end-to-end agentic RAG pipelines, fine-tuning, modular design, and full-stack AI integration. 𝗔𝗺𝗮𝘇𝗼𝗻 𝗧𝗮𝗯𝘂𝗹𝗮𝗿 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗦𝗲𝗮𝗿𝗰𝗵 Master vector search over structured data by building a natural language search engine for e-commerce products. → Learn how to embed, index, and retrieve relevant product data using semantic search. 𝗟𝗟𝗠 𝗧𝘄𝗶𝗻: 𝗬𝗼𝘂𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗥𝗲𝗮𝗱𝘆 𝗔𝗜 𝗥𝗲𝗽𝗹𝗶𝗰𝗮 Create your own LLM-powered twin from scratch—designed to reflect your knowledge and communication style. → Learn fine-tuning, embedding, vector databases, and serving production-grade AI. 𝗛&𝗠 𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗥𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱𝗲𝗿 𝗦𝘆𝘀𝘁𝗲𝗺 Deploy a neural recommender system for fashion items with real-time serving using Hopsworks and KServe. → Learn feature engineering, MLOps, Kubernetes deployment, and retrieval-augmented recsys. Just: → Clone the repo → Open the Substack lesson → Follow the guide + run the code → Remix it, fork it, and make it your own If you’re tired of learning in isolation and want to actually build production AI, these courses are for you. 🔗 Link to all 5 courses: https://lnkd.in/d8gP9cxC

Evaluation is the bottleneck of every serious GenAI system. And 90% of teams are still treating it as an afterthought... If you’re building LLM apps, especially with RAG or agentic systems, you’ve probably hit the same wall: → Messy prompt changes with zero version control → Vector search that “feels” right, but fails silently → Outputs that kinda work, but you have no way to quantify why → No strategy to measure the impact of new features So ahead of my upcoming Open Data Science Conference (ODSC) 2025 webinar, I’m releasing the full open-source evaluation playbook. If you want wants to explore the code before the talk drops, here's your chance... Note, you don’t have to attend the webinar to use it - the README is detailed enough to guide you. Here’s what you’ll get: 𝗠𝗼𝗱𝘂𝗹𝗲 𝟭: 𝗣𝗿𝗼𝗺𝗽𝘁 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 + 𝗩𝗲𝗿𝘀𝗶𝗼𝗻𝗶𝗻𝗴 Track every LLM call and prompt change using Opik by Comet. Visualize agent traces, compare versions, and finally debug with confidence. 𝗠𝗼𝗱𝘂𝗹𝗲 𝟮: 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗥𝗔𝗚 Use UMAP/t-SNE to visualize embeddings. Compute retrieval recall/precision with LLM-as-a-judge. 𝗠𝗼𝗱𝘂𝗹𝗲 𝟯: 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻-𝗟𝗲𝘃𝗲𝗹 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 Detect hallucinations, moderation issues, and quality drops with custom judges. Log everything into Opik to track iterations across builds. 𝗠𝗼𝗱𝘂𝗹𝗲 𝟰: 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗻𝗴 𝗥𝗲𝗮𝗹 𝗨𝘀𝗲𝗿 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 Capture structured feedback from users to fuel future eval splits or fine-tuning jobs (e.g. preference alignment). 𝙒𝙝𝙮 𝙖𝙢 𝙄 𝙙𝙤𝙞𝙣𝙜 𝙩𝙝𝙞𝙨? Because evaluation is hard. And most teams don’t have a mental model for how to think about these moving parts - let alone code for it. This project brings structure, tooling, and clarity to that chaos. 🔗 Link to the repo in the comments. P.S. If you're joining my session at ODSC, keep it bookmarked, we’ll walk through the full stack live.

Here's why 98% agent demos break after 3 turns: (Hint: it's not because the prompts are bad) 𝗧𝗵𝗲 𝗮𝗴𝗲𝗻𝘁 𝗱𝗼𝗲𝘀𝗻’𝘁 𝗿𝗲𝗺𝗲𝗺𝗯𝗲𝗿 𝘄𝗵𝗮𝘁 𝗷𝘂𝘀𝘁 𝗵𝗮𝗽𝗽𝗲𝗻𝗲𝗱. Without memory, you don’t get reasoning. Without reasoning, you don’t get believable agents... You just get brittle demos that fall apart under pressure. That’s why we made Lesson 3 of the PhiloAgents course all about memory. In this lesson, we cover how memory enables: → Conversational flow via short-term memory → Grounded reasoning via long-term memory (with agentic RAG) → Semantic vs. Episodic vs. Procedural long-term memory → Scalable architecture across threads, users, and interactions → Fast, focused context handling through smart summarization We also break down the critical design choices: → What kind of memory structures you actually need → How to avoid bloated infra with a single vector DB → Why long-term memory ≠ just sticking a RAG on top Interested? Lesson 3 is now live. You’ll build all this directly into a philosopher NPC simulation. (Link in comments) P.S. Huge thanks to Miguel Otero Pedrido for the collab on this one. This was one of the most fun to build and it’s a piece most agentic builders overlook.

Writing a book felt like a gamble. But looking back, it was one of the best decisions I’ve ever made. As of today, the 𝗟𝗟𝗠 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿’𝘀 𝗛𝗮𝗻𝗱𝗯𝗼𝗼𝗸 has: - Sold 12,000+ copies - Become an Amazon bestseller - Given me the freedom to build without pressure When I completely denounced my social life to focus on writing - I didn’t know if anyone would read it. I didn’t know if it would open any doors. I didn’t know if it would be worth the effort. Fortunately, it all paid off. The book gave me breathing room to focus, reinvest, and go all-in on what I love: → Content → AI & Software → Building Decoding ML But the impact went far beyond the numbers... →It gave me the confidence that my content is good → It led to speaking invites at QCon, ODSC, and DataCamp → It connected me to incredible collaborators like [@whats-ai]—which sparked our next course on agents → And it directly led to my current consulting role (plus many more I’ve had to turn down) In short: it’s been the catalyst for almost everything I’m building today. I'm extremely grateful to Maxime Labonne for co-authoring this journey and Gebin George for trusting me with the opportunity. TL;DR: If you’re thinking about writing a book, do it. You’re not just publishing words... You’re publishing proof of who you are and what you stand for.

Here's the best piece of advice you need to build real-world agents: "Stop thinking in prompts; start thinking in graphs." Because under the hood, serious agentic systems aren’t just string manipulation. They’re structured, dynamic workflows. And that’s exactly how the 𝗣𝗵𝗶𝗹𝗼𝗔𝗴𝗲𝗻𝘁 works... It’s not a prompt wrapped in a Python script. It’s a full agentic RAG system - Let’s break it down... We use a stateful execution graph to drive our philosopher NPCs. Here’s how: 𝟭. 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗡𝗼𝗱𝗲 Handles the primary logic. It merges incoming messages, current state, and philosopher identity (style, tone, perspective) to generate the next reply. 𝟮. 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗧𝗼𝗼𝗹 𝗡𝗼𝗱𝗲 If the agent needs more information, it calls a MongoDB powered vector search to fetch relevant facts about the philosopher's life and work. This turns simple RAG into agentic RAG, since the LLM dynamically chooses tool calls. 𝟯. 𝗦𝘂𝗺𝗺𝗮𝗿𝗶𝘇𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗡𝗼𝗱𝗲 We summarize long-retrieved passages before injecting them into the prompt. This keeps prompts clean and focused, avoiding dumping in whole Wikipedia pages. 𝟰. 𝗦𝘂𝗺𝗺𝗮𝗿𝗶𝘇𝗲 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗡𝗼𝗱𝗲 If the conversation gets long, we summarize and trim earlier messages and keep only recent context, while preserving meaning. We need the summary for the agent to be consistent and reference early topics from the conversation. This helps keep the context window short and focused, lowering costs and latency and improving accuracy. 𝟱. 𝗘𝗻𝗱 𝗡𝗼𝗱𝗲 Wraps up the cycle. Memory is updated, context evolves, and the agent grows with every message. 𝗛𝗲𝗿𝗲'𝘀 𝘁𝗵𝗲 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗱𝗲𝘁𝗮𝗶𝗹𝘀: - The short-term memory is kept as a Pydantic in-memory state: the PhilosopherState - Tool orchestration with LangChain - Low-latency LLMs, such as Llama 70B, served by Groq - Smaller 8B models used for summarization tasks - Prompt templates are dynamically generated per philosopher - Served as a real-time REST API through FastAPI & WebSockets to power the game UI - Monitoring + evaluation wired through Opik by Comet 𝗜𝗻 𝘀𝗵𝗼𝗿𝘁: Agents come alive through structure, memory, tools, and flow control. You can adapt this exact system to build - → Context-aware assistants → Multi-turn RAG copilots → NPCs, tutors, or internal tools that think and retrieve We walk through every step (with code) in Lesson 2 of the 𝗣𝗵𝗶𝗹𝗼𝗔𝗴𝗲𝗻𝘁𝘀 course. 🔗 Link to the comments

Here's the most annoying thing about MLOps pipelines: (And it's contrary to popular belief) Most break at the last mile. Not during training. Not during evaluation. But at the moment of deployment, more specifically, when testing your local models in production. It's the part where: - DevOps gets looped in late - ML engineers get blocked by infra - Debugging takes forever And worst of all? You might wait 20 minutes just to find out your endpoint doesn’t work. The long cycles to test your ML deployments kill productivity and, most importantly, the inspiration and experimentation speed, critical to building AI solutions. But there’s a better way to approach this... Instead of treating deployment as someone else’s problem, ML teams can take control by testing their models locally before handing them off. Here’s what that looks like in practice: 1. Train and log your model using MLflow 2. Wrap it with a custom class that defines your prediction logic (e.g. convert labels to readable outputs) 3. Download the model artifact using MLflow’s CLI 4. Serve it locally using the MLflow inference server 5. Test the /invocations endpoint with real requests to ensure contract correctness 6. Validate edge cases (e.g. malformed input) to catch failures early This flow ensures your deployment logic works **before** involving production infra, speeding up the development cycle by 10x. A simple shift in mindset. A massive win in practice. Thanks to Maria Vechtomova and Başak Tuğçe Eskili for outlining this workflow so clearly in their latest article. P.S. I highly recommend their course End-to-end MLOps with Databricks.

If you're thinking about consulting in AI, think twice. Here's what it looks like: In week 1 with one of my existing clients shared the product vision and roadmap then dropped a task list that said: → Deploy the product to AWS with CI/CD → Support multiple deployment modes → Add LLM observability → Optimize and stabilize the core system. That’s it. No onboarding, no long handovers. From there, it was all on me to: → Reverse-engineer the code and architecture → Identify missing pieces in the infra → Learn whatever tool or system I needed on the fly → Ship fast in an environment with low resources and zero handholding This is the reality of working as a contractor in early-stage AI teams. You don’t get to ask for the perfect setup. You make decisions in ambiguity. You learn fast, adapt faster, and ship before you feel ready. Here’s what I’ve learned: ✅ You’ll never know everything going in ✅ Mastering fundamentals matters more than mastering tools ✅ You need to balance speed with system thinking ✅ Your job isn’t to follow process—it’s to create one that works under fire If you’re looking to freelance or consult in AI, prepare to be thrown into the fire. 𝗬𝗼𝘂𝗿 𝘃𝗮𝗹𝘂𝗲 𝗶𝘀 𝗶𝗻 𝗵𝗼𝘄 𝗳𝗮𝘀𝘁 𝘆𝗼𝘂 𝗳𝗶𝗻𝗱 𝗰𝗹𝗮𝗿𝗶𝘁𝘆, 𝗻𝗼𝘁 𝗵𝗼𝘄 𝗺𝘂𝗰𝗵 𝘆𝗼𝘂 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝗸𝗻𝗼𝘄. No better prep than building, breaking, and repeating.

The most underestimated part of building LLM applications? Evaluation. Evaluation can take up to 80% of your development time (because it’s HARD) Most people obsess over prompts. They tweak models. Tune embeddings. But when it’s time to test whether the whole system actually works? That’s where it breaks. Especially in agentic RAG systems - where you’re orchestrating retrieval, reasoning, memory, tools, and APIs into one seamless flow. Implementation might take a week. Evaluation takes longer. (And it’s what makes or breaks the product.) Let’s clear up a common confusion: 𝗟𝗟𝗠 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 ≠ 𝗥𝗔𝗚 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻. LLM eval tests reasoning in isolation - useful, but incomplete. In production, your model isn’t reasoning in a vacuum. It’s pulling context from a vector DB, reacting to user input, and shaped by memory + tools. That’s why RAG evaluation takes a system-level view. It asks: Did this app respond correctly, given the user input and the retrieved context? Here’s how to break it down: 𝗦𝘁𝗲𝗽 𝟭: 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. → Are the retrieved docs relevant? Ranked correctly? → Use LLM judges to compute context precision and recall → If ranking matters, compute NDCG, MRR metrics → Visualize embeddings (e.g. UMAP) 𝗦𝘁𝗲𝗽 𝟮: 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻. → Did the LLM ground its answer in the right info? → Use heuristics, LLM-as-a-judge, and contextual scoring. In practice, treat your app as a black box and log: - User query - Retrieved context - Model output - (Optional) Expected output This lets you debug the whole system, not just the model. 𝘏𝘰𝘸 𝘮𝘢𝘯𝘺 𝘴𝘢𝘮𝘱𝘭𝘦𝘴 𝘢𝘳𝘦 𝘦𝘯𝘰𝘶𝘨𝘩? 5–10? Too few. 30–50? Good start. 400+? Now you’re capturing real patterns and edge cases. Still, start with how many samples you have available, and keep expanding your evaluation split. It’s better to have an imperfect evaluation layer than nothing. Also track latency, cost, throughput, and business metrics (like conversion or retention). Some battle-tested tools: → RAGAS (retrieval-grounding alignment) → ARES (factual grounding) → Opik by Comet (end-to-end open-source eval + monitoring) → Langsmith, Langfuse, Phoenix (observability + tracing) TL;DR: Agentic systems are complex. Success = making evaluation part of your design from Day 0. We unpack this in full in Lesson 5 of the PhiloAgents course. 🔗 Check it out here: https://lnkd.in/dA465E_J

95% of agents never leave the notebook. And it’s not because the code is bad... It’s because the system around them doesn’t exist. Here's my point: Anyone can build an agent that works in isolation. The real challenge is shipping one that survives real-world conditions (e.g., live traffic, unpredictable users, scaling demands, and messy data). That's exactly what we tackled in 𝗟𝗲𝘀𝘀𝗼𝗻 𝟭 𝗼𝗳 𝘁𝗵𝗲 𝗣𝗵𝗶𝗹𝗼𝗔𝗴𝗲𝗻𝘁𝘀 𝗰𝗼𝘂𝗿𝘀𝗲. We started by asking, "What does an agent need to survive in production?" And decided on 4 things - It needs an LLM to run in real-time. A memory to understand what just happened. A brain that can reason and retrieve factual information. And a monitor to ensure it all works under load. So we designed a system around those needs. The frontend is where the agent comes to life. We used Phaser to simulate a browser-based world. But more important than the tool is the fact that this layer is completely decoupled from the backend. (so game logic and agent logic evolve independently) The backend, built in FastAPI, is where the agent thinks. We stream responses token-by-token using WebSockets. All decisions, tool calls, and memory management happen server-side. Inside that backend sits the agentic core - a dynamic state graph that lets the agent reason step-by-step. The agent is orchestrated by LangGraph and powered by Groq for real-time inference speeds. It can ask follow-up questions, query external knowledge, or summarize what’s already been said (all in a loop). When the agent needs facts, it queries long-term memory. We built a retrieval system that mixes semantic and keyword search, using cleaned, de-duplicated philosophical texts crawled from the open web. That memory lives in MongoDB and gets queried in real time. Meanwhile, short-term memory tracks the conversation thread across turns. Without it, every new message would be a reset. With it, the agent knows what’s been said, what’s been missed, and how to respond. But here’s the part most people skip: observability. If you want to improve your system, you need to see and measure what it's doing. Using Opik (by Comet), we track every prompt, log every decision, and evaluate multi-turn outputs using automatically generated test sets. Put it all together and you get a complete framework that remembers, retrieves, reasons, and responds in a real-world environment. Oh... and we made the whole thing open source. 🔗 Link: https://lnkd.in/d8-QbhCd P.S. Special shout out to my co-creator Miguel Otero Pedrido