Abide AI

Building a second brain for the authors and creators out there.

Founder

O'Reilly

Authoring book on LLMOps: Managing Large Language Models in Production

1. Report: what is LLMOps ✅
2. Book - LLMOps: Managing LLMs in Production ◻️ (currently writing)

Author

Laconia

Venture Capital Fellow 

 Consultant

Machine Learning Engineer

ASSENT

Machine Learning Scientist

Insight Data Science

Building Data Labelling Pipeline to annotate the data automatically

Data Science Fellow

UCLA Mathematics

Supervisor: Dr. Judea Pearl

Core Expertise: AutoML, Reinforcement Learning, Emotion Recognition in Speech as well as Video (Natural Language Processing, Computer Vision)

Research Scholar - Machine Learning

Amishi Life 

Founding Engineer & Head of Digital - E-commerce

Hidden Creative

Product Development Manager - Augmented Reality Applications

Entrepreneur First

Program Management - Deep Tech

I have over eight years of experience in applying machine learning to various domains, such as e-commerce, insurance, and media & entertainment.

I am also the author of an upcoming O'Reilly book on LLMOps: Managing Large Language Models in Production, where I share my insights and best practices on how to handle the challenges and opportunities of deploying state-of-the-art large language models. 

My mission is to help machine learning practitioners and engineers build more efficient, reliable, and scalable Ops systems.

So, I am sure most of us AI/ML engineers have atleast once said to ourselves "𝘐 𝘸𝘪𝘴𝘩 𝘐 𝘤𝘰𝘶𝘭𝘥 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥 𝘩𝘰𝘸 𝘗𝘺𝘛𝘰𝘳𝘤𝘩 𝘸𝘰𝘳𝘬𝘦𝘥" ⁣⁣⁣

I spent the last month tracing every line from model.generate() down to the final .cubin and t𝐡𝐞 𝐫𝐞𝐬𝐮𝐥𝐭 𝐢𝐬 𝐭𝐡𝐢𝐬 𝐝𝐢𝐚𝐠𝐫𝐚𝐦⁣⁣⁣⁣⁣. I mapped all the 40+ components you need to understand the complete engine behind LLM inferencing.
⁣⁣⁣
⁣⁣⁣⁣⁣Well, h𝐞𝐫𝐞'𝐬 the 𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞 𝐏𝐲𝐓𝐨𝐫𝐜𝐡’𝐬 𝐈𝐧𝐭𝐞𝐫𝐧𝐚𝐥 𝐄𝐱𝐞𝐜𝐮𝐭𝐢𝐨𝐧 𝐦𝐚𝐩 as of 2025

Feel free to 𝐑𝐞-post, share 𝐢𝐭 𝐰𝐢𝐭𝐡 𝐭𝐡𝐨𝐬𝐞 𝐢𝐧𝐭𝐞𝐫𝐞𝐬𝐭𝐞𝐝 𝐚𝐧𝐝 𝐬𝐚𝐯𝐞 𝐢𝐭 𝐬𝐨 𝐲𝐨𝐮 𝐜𝐚𝐧 𝐜𝐨𝐦𝐞 𝐛𝐚𝐜𝐤 𝐭𝐨 𝐢𝐭 𝐥𝐚𝐭𝐞𝐫.⁣
⁣
Some things this chart highlights, details that don't get discussed often or people get wrong quite a lot-

1️⃣ fullgraph=True only works for prefill, coz it breaks during decoding from control flow and dynamic shapes that differ for each iteration. Hence, only the prefill step benefits from full-graph capture.

2️⃣ Dynamo cache size is 64 by default, not 1024. Many think it’s 1024 because that was an earlier internal proposed value. Increasing it may improve performance for workloads that recompile often, but at the cost of memory.

3️⃣ What stage do Inductor kernel fusion operations happen i.e. after the FX graph lowering but before backend compilation.

4️⃣ How FlashAttention works - this Triton → PTX → ptxas → .cubin. This is where most of the LLM inference speed comes from.

5️⃣ ATen dispatch and low-level memory - this enables fast mixed-precision operations. It’s why PyTorch can dynamically switch between FP16/BF16/FP32 without explicit user logic. 

6️⃣ CUDA Graphs breaking on every token. This is where fameworks (like vLLM, TensorRT-LLM) mitigate this by preallocating buffers and keeping shapes static

7️⃣ KV cache memory limitations - this is where OOM errors happen. This is the main bottleneck, especially for FP16 caches.

𝐓𝐡𝐞 𝐫𝐞𝐝 𝐛𝐨𝐱 is 𝐭𝐡𝐞 𝐞𝐱𝐚𝐜𝐭 𝐩𝐥𝐚𝐜𝐞 𝐏𝐲𝐓𝐨𝐫𝐜𝐡 hits 𝐛𝐨𝐭𝐭𝐥𝐞𝐧𝐞𝐜𝐤𝐬 𝐢𝐧 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧.⁣⁣

⁣⁣Everything below the Framework Abstractions line (vLLM, DeepSpeed, Megatron, TensorRT-LLM) exists only to work around these PyTorch limitations.

If you’re:
- debugging why your models throw OOM 
- writing custom kernels
- trying to understand why vLLM is 100× faster than `torch.compile`  
…this is the map you need to understand. 

And ofc, needless to say that I will explain every component in pain-staking detail on a blogpost on 𝐌𝐨𝐝𝐞𝐥𝐂𝐫𝐚𝐟𝐭 (Substack) 👉 https://lnkd.in/gCTieFXB

The next two posts on ModelCraft will be on full stack inferencing (the workflow I shared a couple days ago) and on multi agent systems.

Abi Aryan 🦉

ML Engineer | Venture Capital | Writing LLMOps book ✍️

Founder: Abide AI | Author: LLMOps & GPU Engineering | Making AI Systems go brrrr...

Jitendra Kasaudhan

Its end of 2025 and I still keep getting asked the same question: 𝘈𝘣𝘪, 𝘸𝘩𝘢𝘵'𝘴 𝘵𝘩𝘦 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘤𝘦 𝘣𝘦𝘵𝘸𝘦𝘦𝘯 𝘔𝘓𝘖𝘱𝘴 𝘢𝘯𝘥 𝘓𝘓𝘔𝘖𝘱𝘴? 

I answered in my book, 𝐋𝐋𝐌𝐎𝐩𝐬: 𝐌𝐚𝐧𝐚𝐠𝐢𝐧𝐠 𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 𝐢𝐧 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧(O'Reilly) that came out earlier this year (𝘺𝘰𝘶 𝘤𝘢𝘯 𝘨𝘦𝘵 𝘪𝘵 𝘧𝘳𝘰𝘮 𝘈𝘮𝘢𝘻𝘰𝘯, 𝘴𝘦𝘢𝘳𝘤𝘩 "𝘓𝘓𝘔𝘖𝘱𝘴 𝘣𝘰𝘰𝘬") but hey, we live in the age of viral cat-videos, 𝐬𝐨 𝐈 𝐰𝐨𝐧'𝐭 𝐞𝐱𝐩𝐞𝐜𝐭 𝐲𝐨𝐮 𝐰𝐨𝐮𝐥𝐝 𝐡𝐚𝐯𝐞 𝐫𝐞𝐚𝐝 𝐦𝐲 𝐛𝐨𝐨𝐤. So, let me explain again to bring the point home, once and for all.

𝐓𝐫𝐚𝐝𝐢𝐭𝐢𝐨𝐧𝐚𝐥 𝐌𝐋𝐎𝐩𝐬 (𝐜𝐢𝐫𝐜𝐚 𝟐𝟎𝟏𝟖–𝟐𝟎𝟐𝟑):⁣⁣⁣⁣
⁣⁣⁣ 🌚 Your model size was ~300 MB – 2 GB
⁣⁣⁣ 🌚 Inference cost was about $0.0001 – $0.002 per prediction
⁣⁣⁣ 🌚 50–200 ms was acceptable latency
⁣⁣⁣ 🌚 The data was mostly tabular + a bit of images, nicely structured
And failure meant wrong number which was sad but rarely catastrophic

𝐋𝐋𝐌𝐎𝐩𝐬 𝐢𝐧 𝟐𝟎𝟐𝟓:⁣⁣⁣⁣⁣⁣
😨 Your model size is 8B → 405B params (70–800 GB VRAM when quantised)
😨 Your Inference cost is $0.15 – $12 per 1M tokens (yes, a single long user session can cost you $3–$8 on Grok-4/Claude-3.5/GPT-4o)
😨 Users bail if latency >1.8 s end-to-end 
😨 And you got 100 TB+ of unstructured data, constantly drifting text/code/logs, 90 % of which is garbage you still have to chunk and embed

And failure means that model starts role-playing as a therapist and leaks PII. Lawyers soon on your speed-dial before your kids.

That’s why the LLMOps engineer you hire tomorrow can’t be the same MLOps person you had yesterday. 

𝐈𝐟 𝐲𝐨𝐮 𝐡𝐚𝐝 6 𝐰𝐞𝐞𝐤𝐬 𝐭𝐨 𝐮𝐩𝐬𝐤𝐢𝐥𝐥 𝐭𝐡𝐞𝐦, 𝐡𝐞𝐫𝐞'𝐬 𝐰𝐡𝐚𝐭 𝐈 𝐰𝐨𝐮𝐥𝐝 𝐝𝐨:

✅ 2 weeks paired with an AI engineer just slamming prompts into Llama-3.1-405B, Mixtral-141B, Claude-3.5, Grok-4, whatever. 

Raw, no LangChain guardrails. Let them feel the $400 bill after one afternoon of “let’s try 128k context”.

✅ 2 weeks glued to a data engineer building the actual RAG pipeline, watching 500M chunks get embedded on 4×H100s.

Learning why your pinecone bill went from $400 to $18k in one month, and why latency jumps from 180 ms to 4.7 s the moment you turn on MMR reranking.

Tell them: Here’s $25k credit on Fireworks/Together/Groq, make it last the quarter while keeping p95 latency <1.2 s 

✅ Final 2 weeks, mandate to benchmark at least 4 inference providers + self-hosted vLLM + TensorRT-LLM on the same prompt set and present the cost/latency/quality Pareto frontier to the CTO. 

They need to internalize the tradeoffs. Let them ship (and break) logging + guardrails in prod. One forgotten OpenAI “you are a helpful assistant” default and you’re suddenly paying $11k/day in tokens. 

𝐂𝐨𝐧𝐠𝐫𝐚𝐭𝐮𝐥𝐚𝐭𝐢𝐨𝐧𝐬, 𝐲𝐨𝐮 𝐠𝐨𝐭 𝐚𝐧 𝐋𝐋𝐌𝐎𝐩𝐬 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫 𝐨𝐧 𝐲𝐨𝐮𝐫 𝐭𝐞𝐚𝐦. And the same, applies for your AI/ML Engineers or Member of Technical Staff to upskill them.

Abi Aryan ☯︎

It was a pleasure chatting with Ben Lorica 罗瑞卡 for an episode of the O'Reilly Radar podcast on LLMOps, thank you Nicole Butterfield for inviting me, where I talked about everything LLMOps - Evals, Data Engg, Agent Frameworks, Distributed Systems. Give it a listen, I'm sure y'all would enjoy:

https://lnkd.in/gFwenq_k

Also, I will be speaking about agentic systems today at the Nexus conference by Packt, if you're attending :)

taplio

Abi Aryan ☯︎𓁿's Linkedin Analytics

Abi Aryan ☯︎𓁿

Performance ScoreAbi Aryan ☯︎𓁿's LinkedIn Performance

Abi Aryan ☯︎𓁿's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Abi Aryan ☯︎𓁿's Linkedin Analytics

Abi Aryan ☯︎𓁿

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Abi Aryan ☯︎𓁿's LinkedIn Performance

Abi Aryan ☯︎𓁿's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Abi Aryan ☯︎𓁿's Linkedin Analytics

Performance ScoreAbi Aryan ☯︎𓁿's LinkedIn Performance

Abi Aryan ☯︎𓁿's Best Posts (last 30 days)