Paul Iusztin

🤖 Senior Machine Learning Engineer • MLOps • Contractor ~ Helping startups engineer production-ready ML/AI systems.

I'm the author of the bestseller LLM Engineer's Handbook, lead instructor of the Agentic AI Engineering course, founding AI Engineer of a San Francisco start-up, and obsessed with making knowledge accessible through AI.

With over 10 years of experience in AI and software and 20 apps shipped, I teach AI Engineering as I wanted to at the beginning of my career. End-to-end. From idea to production. From data collection to deploying, monitoring and evaluation. With a focus on AI principles, software patterns and infrastructure systems that will thrive in a future dominated by AI coding tools.

My ultimate goal is to help other engineers escape PoC purgatory and 10x their AI Engineering skills.

.

If you want to become an AI Engineer pro, join Decoding AI for exclusive content on designing, building, and shipping AI software that works: 

🗞️ https://www.decodingai.com

See more:

🌐 Homepage: https://www.pauliusztin.ai
📚 LLM Engineer's Handbook: https://www.amazon.com/LLM-Engineers-Handbook-engineering-production/dp/1836200072/
🤖 Agentic AI Engineering Course: https://academy.towardsai.net/courses/agent-engineering?ref=b3ab31

.

✉️ For collabs, contact me at: p.b.iusztin@gmail.com

Senior Machine Learning Engineer

🟡 Scalable ML batch infrastructure for continuous training & inference

The client, a generative AI leader, struggled with manual training and calling their ML models. As a tedious and unscalable process, they needed to automate their ML processes completely and integrate them with their current web platform.

➞ 𝐃𝐞𝐬𝐢𝐠𝐧𝐞𝐝 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a modular 𝐏𝐲𝐭𝐡𝐨𝐧 𝐩𝐚𝐜𝐤𝐚𝐠𝐞 that composes the ML model required steps into 3 main batch pipelines that can easily be automated: feature, training, and inference, 𝐞𝐥𝐢𝐦𝐢𝐧𝐚𝐭𝐢𝐧𝐠 a 𝐭𝐞𝐝𝐢𝐨𝐮𝐬 𝟐-𝐡𝐨𝐮𝐫 𝐦𝐚𝐧𝐮𝐚𝐥 𝐩𝐫𝐨𝐜𝐞𝐬𝐬/𝐯𝐢𝐝𝐞𝐨.
➞ 𝐃𝐞𝐩𝐥𝐨𝐲𝐞𝐝 a scalable & cost-effective 𝐛𝐚𝐭𝐜𝐡 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 and 𝐂𝐈/𝐂𝐃 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 on top of AWS ECS & SQS that scales in and out based on the number of jobs, 𝐬𝐚𝐯𝐢𝐧𝐠 𝐭𝐡𝐨𝐮𝐬𝐚𝐧𝐝𝐬 𝐨𝐟 𝐝𝐨𝐥𝐥𝐚𝐫𝐬/𝐦𝐨𝐧𝐭𝐡.
➞ 𝐒𝐚𝐯𝐞𝐝 𝐦𝐨𝐧𝐭𝐡𝐬 𝐨𝐟 𝐟𝐮𝐭𝐮𝐫𝐞 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 by using good software practices to make the Python module & batch cloud infrastructure flexible and composable.
➞ 𝐁𝐮𝐢𝐥𝐭 a 𝐯𝐨𝐢𝐜𝐞 𝐬𝐲𝐧𝐭𝐡𝐞𝐬𝐢𝐬 𝐏𝐨𝐂 in ~1 week required to 𝐜𝐥𝐨𝐬𝐞 the 𝐝𝐞𝐚𝐥 with one of the 𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐜𝐥𝐢𝐞𝐧𝐭𝐬.

⚫️ Big data ingestion & archiving pipeline

The client wanted to expand its platform features by securely and safely ingesting, preprocessing, and storing TB of data into their platform.

➞ 𝐂𝐨𝐧𝐭𝐫𝐢𝐛𝐮𝐭𝐞𝐝 to the 𝐝𝐞𝐬𝐢𝐠𝐧 and 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 of a 𝐝𝐚𝐭𝐚 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports 𝐢𝐧𝐠𝐞𝐬𝐭𝐢𝐧𝐠 >𝟓 𝐓𝐁 of data in 𝐨𝐧𝐞 𝐠𝐨 in a reliable, fast, and secure way.
➞ 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐞𝐝 a scalable 𝐝𝐚𝐭𝐚 𝐩𝐫𝐞𝐩𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 that supports multiple media types (images, videos, audio, 3D objects) and creates small preview files that are cheap to transfer, store and visualize. The 𝐨𝐫𝐢𝐠𝐢𝐧𝐚𝐥 𝐟𝐢𝐥𝐞𝐬 are 𝐚𝐫𝐜𝐡𝐢𝐯𝐞𝐝 on an affordable storage solution 𝐫𝐞𝐝𝐮𝐜𝐢𝐧𝐠 𝐜𝐨𝐬𝐭𝐬 𝐛𝐲 𝐮𝐩 𝐭𝐨 𝟖𝟏%.

I'm currently designing a second brain AI agent.

Here's a high-level architecture (steal it and build your own):

The goal is to create an agent that can reason over everything I’ve ever written, watched, or saved while automatically picking up facts, events, and preferences about me.

The agent should then utilize that memory to assist me in writing, thinking, and creating.

At the foundation of everything happening is GraphRAG.

Then we have 3 steps:

𝟭/ 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 = 𝗺𝗲𝗺𝗼𝗿𝘆

All notes, documents, and links flow through a GraphRAG ingestion pipeline:
 
• Extract entities and relationships
• Summarize and embed content
• Store it as a knowledge graph

This gives me three retrieval modes: semantic search, metadata search, and graph traversal.

That’s how the agent can respond to queries like, “Write a section on building a modular MCP server.”

→ Find MCP-related notes
→ Traverse connected ideas 1–2 hops away.

We then orchestrate the entire pipeline using Prefect, which gives us durable workflows with scheduling, retries, monitoring, and failure recovery.

𝟮/ 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹

On top of that memory sits the agent.

It can:
 
• Query the knowledge graph
• Update long-term memory (episodic, semantic, procedural)
• Call external tools like web search or image generation
• Use a smaller fine-tuned “LLM twin” to match writing style

The agent automatically learns my preferences, facts, and events and writes them back into the knowledge graph.

So it never reasons over everything.
It retrieves only the relevant slice of memory.

This inference loop is also orchestrated with Prefect, this time to make agent calls durable: retries, monitoring, and scalable inference.

𝟯/ 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝘃𝗶𝗮 𝗠𝗖𝗣

The agent is exposed as an MCP server using FastMCP (by Prefect).

This lets me interact with it directly from tools like Claude or Cursor, refining ideas conversationally instead of relying on one-shot prompts.

𝗧𝗵𝗲 𝗰𝗼𝗿𝗲 𝗶𝗱𝗲𝗮 𝗶𝘀 𝘁𝗵𝗶𝘀:

Effective agents are built with structured memory + reliable orchestration.

P.S. Have you tried to build a Second Brain Agent yet?

Senior AI Engineer • Founder @ Decoding AI • Author @ LLM Engineer’s Handbook ~ I ship AI products and teach you about the process.

If you’re building AI agents in 2026,

You need a faster way to make architecture decisions...

95% of agents never hit production.

And it's because teams pick the wrong architecture from day one.

They use workflows when they need an agent.
Multi-agent when a simple pipeline would have worked.
Autonomy where determinism was enough.

So we created something to help with that...

Introducing, Building AI Agents Architecture Cheat Sheet.

It was created by Louis-François Bouchard from Towards AI, Inc. in collaboration with me for our Agentic AI Engineering course that's coming this month.

The idea was to help you make the best architecture decisions.

Specifically, here's what it helps you do:
 
• Decide Workflow vs Single Agent vs Multi-Agent based on real constraints
• Avoid unnecessary autonomy that explodes cost, latency, and debugging time
• Apply the Autonomy Test to know when you actually need an agent
• Design thin agents + heavy tools for reliability
• Add validation loops, observability, and human in the loop (HITL) by design
• Pick the minimum architecture that works before scaling complexity

It also includes:
 
• A practical kickoff decision process
• Architecture questions with concrete build outcomes
• Clear signals for when multi-agent systems actually make sense
• Production rules we use when shipping real agentic systems

This is exactly the kind of reference you want before you write any code.

If you want it, here's how to get it...
 
Link: https://lnkd.in/eb_V3ck7

P.S. Huge credit to Louis-François Bouchard and the Towards AI, Inc. team for putting this cheatsheet together.

I noticed something interesting after publishing LLM Engineers Handbook:

AI engineering is still widely misunderstood.

Many people expected the book to be a step-by-step guide to fine-tuning LLMs.

And this makes sense to an extent...

Fine-tuning is the most visible part of the AI stack.

But this is not what makes you an AI engineer.
It's just part of the stack.

In fact, it's a small commoditized part of the snack.

These days, frameworks like Unsloth AI, Axolotl, and TRL from Hugging Face make fine-tuning relatively straightforward.

The real engineering work is about:
 
• Designing data pipelines that reliably collect, clean, and transform data
• Building infrastructure that scales, recovers, and stays observable
• Orchestrating RAG pipelines, agents, and workflows
• Managing datasets, lineage, evaluations, and monitoring in production

In other words, data is the bottleneck.

If you fine-tune on a random dataset and run a cookbook script, you have learned the “hello world” of fine-tuning.

The challenge is knowing what data you need, how to get it, how to structure it, and how to keep it fresh as systems evolve.

So if you want to grow as an AI or LLM engineer, your focus must shift.

Stop asking, “How do I fine-tune this model?”

Start asking, “How do I build a system that feeds the model the right data, at the right time, reliably?”

This is where production systems succeed or fail

Model architecture tweaks and novel training tricks are important, but that work sits closer to research than engineering.

Both matter.
They are just different jobs.

AI engineering is about shipping systems that work, scale, and survive real users and real data.

And once you see that clearly, your learning priorities change completely.

GraphRAG is becoming mandatory for shipping real-world AI agents.

Here’s why:

Shipping reliable AI agents is mostly a data and memory problem.

But the majority of engineers still treat it like a prompting issue.

An agent fails, and the instinctive reaction is:

“Rewrite the prompt.”
“Add more instructions.”
“Increase the context window.”

Sometimes that helps.
Most of the time, it doesn’t.

Because modern agents rarely fail due to a lack of reasoning power...

They fail because they’re forced to reason over too much, too little, or the wrong context.

As systems grow, context fragments:
 
• Knowledge lives across docs, tickets, dashboards, and chat logs
• Ownership and dependencies exist only in people’s heads
• Old workarounds outlive their original intent
• Stale or irrelevant data quietly pollutes the context window

Longer context windows didn’t fix this...
They just made the problem easier to hide.

Performance degrades.
Contradictions creep in.
The agent repeats itself, hallucinates, or acts on outdated assumptions.

This has a name: context rot.

The real questions shouldn't be towards the prompts.

They should be targeted at the architecture:
 
• How do you store organizational knowledge long-term?
• How do you retrieve only what’s relevant right now?
• How do you prevent bad data from contaminating the context window?
• How do you connect memory, ownership, dependencies, and history?

This is where knowledge graphs and GraphRAG become essential.

Instead of dumping documents into a prompt, you model your domain: services, teams, incidents, decisions (and how they relate).

This is exactly what I'll walk through at AI Superstream: Context Engineering by O’Reilly.

Specifically, we'll cover:
 
• When vanilla RAG is not enough
• How knowledge graphs act as long-term memory for agents
• How GraphRAG keeps context windows small but meaningful
• How to design agent memory that scales in real systems

If you’re building agents meant to survive real users, real data, and real time, this layer is non-negotiable.

Prompting is the surface.
Context engineering is the foundation.

📆: 26th February 2026

🔗: https://lnkd.in/exndWkcS

P.S. Seats are limited.

P.S.S. Huge thank you to Angelina Yang for inviting me to this panel and to O'Reilly for making this event possible.

Most production incidents aren’t hard to fix.

But they are hard to understand.

By the time an alert fires, the problem usually isn’t new…

Someone has seen it before.
There’s a workaround.
There’s history.

But that context is scattered across Slack threads, old runbooks, dashboards, and people’s heads.

As systems grow, this becomes the real failure mode.

Not broken code.
Broken memory.

And this is exactly why GraphRAG matters in production systems.

Without graph-based memory, engineers lose time reconstructing what the organization already knows, instead of fixing the issue at hand.

The way out isn’t more alerts or better dashboards…

It’s designing agents that carry organizational memory.

In the latest episode of Decoding AI Magazine, I walk through how to design a Production Engineer agent that reacts to alerts by:
 
• Identifying affected services and teams
• Understanding how failures propagate through dependencies
• Surfacing the context engineers usually spend hours digging for

The key insight is memory.

We use GraphRAG to model services, teams, incidents, and documentation as a connected system the agent can traverse when it matters most.

If you’re building agentic systems for production environments, this is the mental model that holds up at scale.

The full breakdown will go live tomorrow at 1 PM CET.

Subscribe to Decoding AI Magazine to get it delivered to your inbox.

Link → https://lnkd.in/et8fmmBQ

I was a true believer in clean architecture.

I honestly thought it would help me structure my Python AI projects.

This resulted in countless hours wasted obsessively dividing my projects into four folders:
 
• Domain
• Application
• Infrastructure
• Interface

It felt right.
But it was a complete nightmare to maintain.

It wasn’t until I realized that these layers should be virtual concepts that I finally cracked it.

Because Python doesn’t force structure on you.
It gives you freedom.

And that freedom is exactly why AI projects fall apart.

If you don’t impose some structure, you get spaghetti code.
If you impose too much structure, you get ceremony and friction.

Most advice pushes you to one extreme or the other.

Either rigid Clean Architecture, copied from Java, or tool-specific templates that lock you into a framework.

Neither worked for me.

What finally worked was treating clean architecture as a set of boundaries.

The domain became “pure business logic.”
The application became orchestration.
Infrastructure became swappable details.
Interfaces became replaceable entry points.

But none of that needed to be enforced dogmatically in folders.

Once I made that shift, things got simpler.

AI agents became reusable.
Workflows became easier to change.
Switching models or storage stopped rippling through the codebase.

If your Python AI project feels messy and fragile, or clean but painful to work in...

You’re probably applying the right ideas in the wrong way.

Btw, I wrote down the exact mental model and project structure that finally worked for me while building AI agents and workflows.

Sharing it here in case it helps someone else…

Link: https://lnkd.in/eGsU7ww3

P.S. If clean architecture ever made your Python project worse before it made it better, you’re not alone :)

I created an AI agent to write a Substack post (and it went viral).

Nobody even noticed it was AI. The secret?

How we evaluated the agent.

Most people think creating accurate agents is about prompt magic, but that's not the case.

Your evaluation process can make or break everything.

So here's the exact framework we used:

𝟭/ 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲

You can’t design agents or evals in the abstract.

For us, the requirements were explicit:
 
• 10–20 minute technical articles
• Markdown output
• Clear structure and narrative flow
• Mermaid diagrams
• Grounded research

The agent wasn’t allowed to “be creative.”
It was either correct or incorrect.

𝟮/ 𝗧𝗿𝗮𝗻𝘀𝗹𝗮𝘁𝗲 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗺𝗲𝗻𝘁𝘀 𝗶𝗻𝘁𝗼 𝗯𝗶𝗻𝗮𝗿𝘆 𝗰𝗵𝗲𝗰𝗸𝘀

We didn’t use vague 1–5 scores.

We wrote yes/no metrics:
 
• Did the structure match expectations?
• Did it follow the storyline?
• Did it stay grounded in provided research?
• Were diagrams, code blocks, and formatting correct?

Binary metrics force clarity.
They also make finding failures easier and debuggable.

𝟯/ 𝗦𝗲𝗽𝗮𝗿𝗮𝘁𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵 𝗳𝗿𝗼𝗺 𝗻𝗼𝗻-𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵

Some things can be checked directly.
Others can’t.

So we designed two layers:
 
• With ground truth: direct comparison to references
• Without ground truth: style, input coherence, topicality

Both matter.

One anchors the system.
The other preserves flexibility where it’s needed.

𝟰/ 𝗕𝘂𝗶𝗹𝗱 𝗮𝗻 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗷𝘂𝗱𝗴𝗲

Each metric had:
 
• A custom prompt
• 20–30 few-shot examples
• A binary decision
• A written reason

Why?

It tells you why something failed, not just that it did.

𝟱. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝘁𝗵𝗲 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿

We labeled another 20–30 samples ourselves.

Human vs model agreement became the metric.

Once that loop was stable, iteration became engineering:
 
• Swap models
• Push changes
• Score outputs
• Debug with confidence

This framework scales to any agent system.

And it’s exactly the kind of thinking we go deep on in the Agentic AI Engineering course.

We’ll be going live in February 🚀

If you want to learn how to:
 
• Build agents with real evaluation layers
• Design LLM-as-judge systems correctly
• Move from “it works” to “it’s reliable”
 
Join the waitlist here → https://lnkd.in/dtxZuZpb
 
P.S. If you’re building agents without evals, you’re in for a bumpy ride.

In 4 days, I will be live on the DataTalksClub podcast, taking live QAs on AI Engineering, Skill Stack, Agents, LLMOps, and How to Ship AI Products.

Together with Alexey Grigorev we will chat about: 

• The principles behind production-grade systems
• How to build reliable agentic workflows​
• ​The LLMOps practices that keep models stable​
• ​​The patterns I've learned from building more than 20 AI applications

Plus answering all your amazing questions!

The podcast is live, virtual and FREE on 26 Jan.

Join here: https://luma.com/fx3wplwn

See you there ✌️

Thank you Alexey Grigorev for inviting me! So excited about this one.

FastMCP is the best way to build or hook to MCP servers nowadays. At agent-to-tools or even agent-to-agent communication (where an agent is exposed as a tool), MCP is the clear winner. That's why I'm super thrilled to see how quickly FastMCP is adapting to this change with their dynamic features included in their freshly baked 3.0 version.

DAY 1: FastMCP 3.0 Beta is live 🚀

We’re kicking off this week with a rebuild of the FastMCP foundation.

The foundation is rebuilt to move beyond simple tool scripts and into composable AI infrastructure.

𝙼𝚊𝚓𝚘𝚛 𝚄𝚙𝚍𝚊𝚝𝚎𝚜 👇
🔌 Provider Architecture: A unified system that replaces duplicated code across features for flexible, dynamic server building.

👤 Dynamic Scoping: Servers can now show different tools to different users from a single endpoint.

🧩 Composite Apps: Combine tools from data, platform, and product teams into one authorized interface without collisions.

🔍  Context Optimization: New features for search, curation, and progressive disclosure to keep agent context windows clean.

💡 Agent Skills: Direct instructions on how and when an agent should use specific tools.

𝙱𝚊𝚌𝚔𝚠𝚊𝚛𝚍 𝙲𝚘𝚖𝚙𝚊𝚝𝚒𝚋𝚒𝚕𝚒𝚝𝚢: The core Pythonic API remains the same. This rebuild adds the infrastructure required for dynamic environments without breaking your current v2 servers.

🚀 Install the beta: 𝐩𝐢𝐩 𝐢𝐧𝐬𝐭𝐚𝐥𝐥 𝐟𝐚𝐬𝐭𝐦𝐜𝐩==𝟑.𝟎.𝟎𝐛𝟏
💬 Join the Discord: https://lnkd.in/gVH4jZvE
📖 Docs: https://lnkd.in/dCH4cFmA
📝 Announcement Blog: https://lnkd.in/gWx24H_r
✨ What's New Blog: https://lnkd.in/ghQDjqQF

𝒪𝓃ℯ 𝓂ℴ𝓇ℯ 𝓉𝒽𝒾𝓃ℊ.

FastMCP is how you BUILD MCP servers.

Tomorrow we're showing you how you take them to a whole new level. 🌅

As the founder of LLM Watch, my goal is to curate the most relevant Large Language Model (LLM) research and make it accessible to a broad audience. 

I distill complex technical topics into summaries that respect your time. 

Before embarking on this journey, I led Deep Learning R&D at one of Europe's largest telecom companies. There, I spearheaded numerous AI projects with a focus on Natural Language Processing (NLP) and Automatic Speech Recognition (ASR). 

With more than 6 years of experience in AI, I was there before the hype. 

I try to offer a healthy balance between shiny new things and what really matters.

Pascal Biese

Self-Evolving Memory:

A new framework from OPPO AI and LV-NUS lab now introduces memory systems that evolve their own architecture.

The paper draws a compelling analogy to human learning. Most AI agents today are like students who know how to take notes, but always use the same study method regardless of the subject. MemEvolve creates "adaptive learners" that dynamically adjust their learning strategy based on what they're trying to accomplish - memorizing facts for one task, abstracting patterns for another.

For this, they developed a dual-evolution process. The inner loop accumulates experience within a fixed memory architecture. The outer loop does something more interesting: it evaluates how well that architecture is working and proposes structural modifications. 

The system decomposes memory into four modules - Encode, Store, Retrieve, Manage - and uses performance feedback to diagnose bottlenecks and redesign components accordingly. 

The results? MemEvolve improves agent frameworks like SmolAgent and Flash-Searcher by up to 17% on challenging benchmarks. And the memory architectures evolved on one task transfer effectively to unseen benchmarks and different backbone models, suggesting the framework discovers generalizable principles rather than task-specific heuristics. 

In addition to the paper, they also released EvolveLab which unifies twelve existing memory systems under a common interface, making this a practical foundation for future research on self-improving agents. 
↓ 
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

AI Lead at PwC <​/​> Daily AI highlights for 80k+ experts 📲🤗

MemEvolve: Meta-Evolution of Agent Memory Systems

A DeepSeek moment for AI memory? 

DeepSeek AI just published research suggesting that modern language models are burning valuable compute simulating memory lookups through expensive neural operations.

The problem is fundamental: language modeling involves two very different tasks. One is compositional reasoning - connecting ideas, logical inference. The other is knowledge retrieval - recognizing "Diana, Princess of Wales" or common phrases. Current Transformers have no native mechanism to simply look up static patterns, so they're forced to reconstruct them through multiple layers of attention and feed-forward networks.

The proposed solution is called Engram, a "conditional memory" module based on N-gram embeddings that retrieves static patterns in O(1) time. The key insight is that it's about finding the optimal balance between conditional computation (MoE experts) and conditional memory. The researchers discovered a U-shaped scaling law: pure MoE isn't optimal. Reallocating roughly 20-25% of sparse parameters to memory yields superior results. 

While memory was expected to help knowledge-intensive tasks (MMLU +3.4), the largest gains appeared in reasoning (BBH +5.0) and code/math (HumanEval +3.0, MATH +2.4). Mechanistic analysis reveals why: by offloading pattern reconstruction, Engram effectively deepens the network - shallow Engram layers produce representations equivalent to much deeper MoE layers. 

Perhaps most practically significant: the deterministic addressing allows offloading 100B-parameter tables to host memory with less than 3% overhead. 

This suggests that the next generation of sparse models might not just be about smarter routing - but about recognizing that some knowledge simply doesn't need to be computed at all.
↓ 
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

A New Axis of Sparsity for Large Language Models

Meta's Super Intelligence Lab just published their solution to the training data problem. 

Training search-augmented LLMs typically requires extensive human-annotated questions and answers. As high-quality data becomes scarcer and more expensive, this dependency creates a significant bottleneck for developing capable AI research agents.

Dr. Zero introduces a clever proposer-solver feedback loop. Both models start from the same base LLM, but play different roles: the proposer generates increasingly difficult questions, while the solver learns to answer them using external search. 

The key insight is that as the solver improves, easy questions stop providing useful training signal, which forces the proposer to generate harder ones. This creates an automated curriculum that pushes both models forward without any human data.

The computational challenge with such approaches is typically prohibitive. Standard reinforcement learning methods require sampling many questions and many answers per question, creating nested costs that explode for multi-turn reasoning. Dr. Zero solves this through Hop-Grouped Relative Policy Optimization (HRPO), which clusters questions by their structural complexity (number of reasoning "hops" required) and computes advantages within these groups. This reduces sampling overhead to roughly one-fourth of standard methods while maintaining performance. 

The data-free Dr. Zero matches or exceeds fully supervised search agents across seven QA benchmarks, with up to 14.1% improvements on complex multi-hop reasoning tasks. 

This suggests that complex reasoning and search capabilities can emerge purely through self-evolution. 
↓ 
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

Self-Evolving Search Agents without Training Data

LLM-as-a-Judge -> Agent-as-a-Judge

Using LLMs to evaluate other LLMs seemed like a clever shortcut. But single-pass evaluation is hitting its limits: we need evaluators that can actually verify claims, not just assess how plausible an answer sounds. 

Traditional LLM judges are passive observers. They grade based on linguistic patterns without checking if the code runs, the math is correct, or the facts hold up. 

A new comprehensive survey traces the shift to Agent-as-a-Judge, where evaluation becomes an active, multi-step process. Instead of one model making a snap judgment, agentic judges decompose evaluation into subtasks, invoke external tools like code interpreters and search engines to verify claims, and use multi-agent debate to neutralize individual model biases. 

The researchers identify three developmental stages: procedural agents following fixed workflows, reactive agents adapting based on intermediate feedback, and self-evolving agents that can refine their own evaluation criteria on the fly. Applications already span code verification, fact-checking, medical assessment, legal judgment prediction, and financial research analysis. 

When AI systems are used to generate training signals for other AI systems, the precision of the judge determines everything downstream. Can we build evaluators that know when they need to verify rather than guess?
↓ 
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com

A Survey on Agent-as-a-Judge

What if you could surgically edit an LLM's knowledge by swapping a single vector?

A new architecture from CMU and Meta AI makes this possible, while also solving some of the thorniest problems in scaling transformers.

Scaling language models has traditionally meant more compute. Mixture-of-Experts (MoE) promised a workaround by activating only a subset of parameters per token, but came with notorious training instability, load balancing headaches, and communication overhead that grows with expert count.

Instead of dynamic routing to experts, STEM (Scaling Transformers with Embedding Modules) replaces the FFN up-projection with a static, token-indexed embedding lookup. Each token has its own embedding per layer, while the gate and down projections remain shared. No routing decisions, no load balancing losses, no cross-device communication for expert selection. 

The results? 3-4% accuracy gains over dense baselines, with particularly strong improvements on knowledge-intensive benchmarks (9-10% on ARC-Challenge and OpenBookQA). Training remains stable where MoE would show loss spikes, and the architecture naturally activates more parameters as context length grows. 

Because each embedding maps to a specific token, you can literally swap "Spain" for "Germany" at the embedding level and watch the model output "Berlin" instead of "Madrid", without changing the input text at all. 

If interpretability and controllability matter as much as raw performance, STEM suggests we may have been overcomplicating sparse scaling all along.
↓
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com

STEM: Scaling Transformers with Embedding Modules

The next big step in LLMs?

A new inference technique from Massachusetts Institute of Technology called Recursive Language Models (RLMs) is rethinking context windows entirely.

The core problem is well-known: even frontier models like GPT-5 suffer from "context rot" - performance degrades quickly as prompts get longer, regardless of the technical context limit. Summarization helps but loses critical details. Retrieval misses complex reasoning patterns. Have we been trying to make models see more, when perhaps the answer is to make them see differently?

RLMs treats the prompt not as direct neural network input, but as an external object the model interacts with programmatically. The prompt is loaded as a variable in a Python REPL environment, and the LLM writes code to peek into it, decompose it, and recursively call itself over smaller snippets. Same interface as a regular LLM, radically different execution.

On information-dense tasks where GPT-5 scores below 0.1%, RLMs achieve 58%. On multi-hop research questions spanning 6-11M tokens, RLMs hit 91% accuracy while costing less than feeding the full context would. Crucially, performance degrades far more gracefully as complexity scales - the approach handles inputs two orders of magnitude beyond native context windows.

This suggests that scaling context is not just an architecture problem but also an inference problem. RLMs demonstrate that letting models reason about their input symbolically rather than processing it neurally could be a promising new direction.

If this approach generalizes, we may be looking at a new axis for scaling language model capabilities entirely.
↓ 
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

Recursive Language Models

The latest reasoning models fail on puzzles with just 1 unknown cell.

Yet they solve complex ones with dozens of steps.

New research reveals why: these models might be pattern matching, not actually reasoning.

Hierarchical Reasoning Models (HRMs) crush benchmarks, outperforming LLM-based reasoners by wide margins. But researchers dug into the mechanics and found something troubling.

Think of it like a student who memorizes exam answers. They ace questions they've seen before but stumble on trivially easy variations.

The researchers identified three failure modes:

- HRMs fail on extremely simple puzzles (single unknown cell) when patterns violate their training distribution
- Models lack the "fixed point property" - they can't recognize when a problem is already solved
- Performance doesn't correlate with puzzle difficulty the way true reasoning would

This matters because we're deploying these models assuming they generalize.

If your reasoning model solves a 50-step problem but fails a 2-step variant, it's not reasoning - it's sophisticated pattern matching.

Before trusting HRMs in production, test them on deliberately simple edge cases. The failures will tell you more than the successes.
↓
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

taplio

The Battle

Paul Iusztin

Pascal Biese

Followers

Posts

Engagements

Likes

Performance ScorePaul Iusztin's LinkedIn Performance

Performance ScorePascal Biese's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Paul Iusztin

Pascal Biese

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Paul Iusztin's LinkedIn Performance

Performance ScorePascal Biese's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScorePaul Iusztin's LinkedIn Performance