I’ve been working in AI for close to a decade, starting well before today’s LLM-driven wave.

My work is grounded in understanding AI across layers — from hardware, AI models, and data to system design, reliability, and scale. I approach AI as an end-to-end engineering problem: not just modeling, but how models interact with data, infrastructure, and real-world constraints.

Alongside this, I write The AI Merge, a technical newsletter on AI foundations and systems engineering, focused on clear thinking and practical system design rather than hype.

I’m currently building an AI Systems course, centered on end-to-end system design, for engineers who want a deeper, systems-first understanding of modern AI.

🗞️ Read The AI Merge: https://read.theaimerge.com

Alex Razvant

A lot of courses out there teach AI & LLMs, but very few touch on Inference

Sure, it's easy to grasp the fundamentals of what LLMs are and how to build systems around them, but scaling out an LLM out of a lab of 1-2 H100s running it and a single concurrent user session is a big challenge.

Efficient inference is one of the largest barriers to LLMs in production.

Batching, managing memory, concurrent user sessions, GPU saturation, and sharding with/or MIG (Multi-Instance GPUs), parallelisation are key concepts for engineers to know, understand, and work with or around them.

I've had this list of resources saved for a while, which fills some of those gaps.

1. Good Courses / Videos
- CS336 Stanford (Lecture 5-11): https://buff.ly/9o4jRxl
- GPU MODE: https://buff.ly/bPEHYL0 
- LLM Inference Patterns: https://buff.ly/4ASDS5c
- LLM Inference (NVIDIA): https://buff.ly/muDMcor 

2. Repositories / Blogs
- GPU Performance Engineering (E.Andere): https://buff.ly/sfGJNMK 
- AI Performance Engineering (C.Fregly) https://buff.ly/8ey3GPa 
- Inside NVIDIA GPUs (A.Gordic): https://buff.ly/wmgNwMg 
- Understanding LLM Inference (my article): https://buff.ly/NrNU3Qz 

I don't think there's a clear, step-by-step guide on how to master AI Perf Engineering.
The best bet is to study the core pieces of the models and the hardware they run on.

Then, learn the rest on demand.

undefined undefined

Neural Bits CTA

Join 4100+ engineers and get a weekly newsletter on advanced AI/ML. 👉 https://neuralbits.substack.com/

Kubrick

Ready to start building?

--- Resources ---
📌 Kubrick Code Repository
https://github.com/multi-modal-ai/multimodal-agents-course

📌 Neural Bits Newsletter (mine)
https://multimodalai.substack.com/

📌 The Neural Maze Newsletter (Miguel's)
https://theneuralmaze.substack.com/

Senior AI/ML Engineer | Author of NeuralBits Newsletter – helping you build efficient AI Systems

Hard work always pays off, be consistent!

I'm a Senior Machine Learning Engineer with a keen interest in developing AI/ML solutions for real world problems and helping others get started on their Machine Learning Journey.

Let's connect @:
📘 https://medium.com/@alexandrurazvant
✉️ alexandrurazvant@gmail.com
🌐 https://www.neuraleaps.com

Upskill your AI Engineering!

AI Engineering Hub

Senior AI Engineer | Writing The AI Merge Newsletter | Helping Engineers Build AI Beyond Demos

The 5 structure-layers of any LLM Model.
(Understand these, and you'll understand every OSS model on HF)

With each open-source LLM released, you'll find the model weights on Hugging Face.

When you go to the Files section, you won't find a `model.xyz` file, but multiple files, usually 10-15 that compose the entire LLM.

Here's the blueprint to understand them:

1️⃣ 𝘼𝙧𝙘𝙝𝙞𝙩𝙚𝙘𝙩𝙪𝙧𝙚 𝘾𝙤𝙣𝙛𝙞𝙜𝙪𝙧𝙖𝙩𝙞𝙤𝙣 (config.json)
This one contains metadata on model architecture, layer activations, sizes, vocabulary size, number of attention heads, model precision, and more. 

The transformers library knows how to parse this config and build the model architecture. 

2️⃣ 𝙈𝙤𝙙𝙚𝙡 𝙒𝙚𝙞𝙜𝙝𝙩𝙨 (the .safetensors splits)
LLMs have B (billions) of parameters.
For safer download, model weights are split into chunks/shards.
The formats used for weights are either Binary or Safetensors.

Safetensors format is an alternative to the default Pickle serializer that PyTorch (pt) uses, as it’s vulnerable to code injection.

3️⃣ 𝙇𝙖𝙮𝙚𝙧 𝙈𝙖𝙥𝙥𝙞𝙣𝙜 (the *.index.json)
Since the models are large and weights come as part files (e.g., 0001-of-0006, 0002-of-0006, etc.), this file stores a sequential map of the model architecture.

This specifies what layers are part of what model chunk.

4️⃣ 𝙏𝙤𝙠𝙚𝙣𝙞𝙯𝙚𝙧 𝘾𝙤𝙣𝙛𝙞𝙜 (tokenizer.json)
The tokenizer config file contains metadata about the tokenizer used to train this model. 

It contains the class name used to instantiate the tokenizer, the layer names, the structure of how the inputs are processed before passing through the model, and special tokens.

The LLM generally uses 𝘴𝘱𝘦𝘤𝘪𝘢𝘭_𝘵𝘰𝘬𝘦𝘯𝘴 as markers to stop generation.

5️⃣ 𝙂𝙚𝙣𝙚𝙧𝙖𝙩𝙞𝙤𝙣 𝘾𝙤𝙣𝙛𝙞𝙜 (generation_config.json)
These configuration files contain metadata for Inference, such as Temperature and TopP/TopK thresholds or context window size, with which the model was trained. 

When you load your model using Transformers.from_pretrained(), all these files are processed to set up everything you need to actually use that model.

---
Learn AI going through Fundamentals, Engineering and Systems.
Join here (free): https://buff.ly/DnpHEXD

The 4 patterns of LLM Inference.

Deploying LLMs is not the same as deploying traditional ML models. The differences show up most clearly at inference time. 

Some principles such as optimized models, infrastructure, GPU saturation and usage or distributed systems concepts still apply.

But LLM inference adds a new layer of complexity, as Transformer models inference is way more nuanced.

With LLMs, the output is generative and not a fixed size having batch-size input = batch-size output that more traditional ML or Deep Learning models had.

LLM inference splits into two stages, Prefill and Decode, this creates 4 distinct patterns an AI Engineer/Dev should know about before optimizing or deploying.

⇢ In the prefill phase, the model processes the entire input prompt and computes the key-value (KV) cache. In benchmarks, we'll often see this metric as `pp` or `prompt processing` speed, which is fast.

⇢  In the decode phase, it autoregressively generates tokens one at a time, updating the KV cache at each step. In benchmarks, this is refered to as `tg` or text-generation speeds, and this process is memory-bound as tokens have to travel within the GPU's memory during KV cache updates.

Understanding this gives you these four distinct LLM inference patterns:

SISO -- Short Input, Short Output
→ Low latency with fast generation. If using this pattern, aim to optimize for high concurrency.

LILO --  Long Input, Long Output
→ The most compute-intensive requires distributed inference, ideally with shared KV cache across nodes or tensor parallelism.

LISO -- Long Input, Short Output
→ This one has heavy prefill and quick decode. GPUs with enough VRAM would parallelize the prefill and generate fast.

SILO -- Short Input, Long Output
→ Light prefill, which runs fast but is heavier on decode. Decode latency dominates here.

---

𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: The longer the output, the more time is spent in the decode loop.
Before fine-tuning your own LLM and deploying it, also consider the inference pattern your system will target.

Summary:
- In LLM inference, there are 4 patterns.
- For LISO, the generation and TFTT are fast.
- On SISO, everything is fast.
- For LILO,  a lot of computing.
- For SILO, prefill is fast, generation is slow.

---

Learn AI through Fundamentals, Engineering and Systems.
Follow and join here: https://buff.ly/iqFpm9x

If you'll ever try to run LLMs locally - you'll bump into llama.cpp

Most guides tell you what to use for Edge/CPU/low-end GPU inference - but not what’s actually happening under the hood. 

No context on:
- How is the model checkpoint parsed?
- How are tensors unpacked and mapped?
- How do GGML, GGUF, and llama.cpp fit together?

✅ So I went through the codebase myself. 

I spent a few hours digging through llama.cpp, PRs, and community threads, then turned everything into a single sequence diagram that shows the flow end-to-end. 

The key pieces of the stack:
> GGML - the ML Tensor Library and execution primitives (ops, mappings, kv-cache)
> GGUF - the format for storing quantized LLM models and Quantization types.
> llama.cpp - is the C++ inference engine, building on top of GGML & GGUF, wiring everything together.

If you’re running local LLMs, this is the stack you’re relying on.

If you want to understand why things break, how quantized models are actually loaded, this breakdown will help. 

---
I share more practical advice on AI Fundamentals, Engineering and Systems.
Join here (free): https://buff.ly/iqFpm9x

You should read more code.

If you’re already a Software Engineer or AI / ML Engineer, one of the fastest ways to level up right now is to study how big AI teams write and ship - in public. 

Many top companies and major contributors to the AI space have been open-sourcing their internal frameworks, kernels, or infrastructure libraries. 

You might feel overwhelmed by how fast AI is going. But this is a great learning opportunity if you know where to look.

Learning doesn’t just happen from papers and tutorials, especially if you already build software.

You should:
- Read real production code
- Study PRs or commit history on major/popular repos.
- See how teams structure codebases, design APIs, configs, tests

Check what gets reviewed, what gets rejected, and why.
This kind of learning will upgrade your engineering intuition. 

Here are some excellent codebases to learn from.

✅ From DeepSeek:
1. FlashMLA  - A library of high-performance attention kernels.
Link: https://buff.ly/XbFnPA7 

2. DeepEP - A Communication Library for high-throughput MoE workloads.
Link: https://buff.ly/3qxMc3S

3. 3FS - A Distributed Filesystem for AI Workloads, for AI training and inference.
Link: https://buff.ly/ovfW47A

✅ From NVIDIA:
1. Dynamo - A datacenter-scale Inference Serving Framework for GenAI workloads.
Link: https://buff.ly/WyAMzBz 

2. NeMo  -  A Scalable Generative AI Framework, for training and deploying LLMs and VLMs.
Link: https://buff.ly/L9KTuJp 

✅ From vLLM:
1. The vLLM Library  - One of the most popular LLM inference engines, designed for scale.
Link: https://buff.ly/GLCfoIM

✅ From SGLang:
1. The SGLang Engine - A high-performance serving engine for LLMs & multimodal models.
Link: https://buff.ly/m4HhsWQ 

✅ From llama.cpp
1. The llama.cpp Engine - A minimal C/C++ inference engine for LLMs with broad hardware support and quantization options.
Link: https://buff.ly/u9yaXwp

📌 Don't study everything, you won't be able to.

Instead, pay attention to:
⇾ Code structure & abstractions
How are core functionalities separated (core logic vs hardware-specific optimizations)?

⇾  PRs & commit history
What context do authors provide? How do reviewers push back?

⇾  Interfaces & public APIs
How are public APIs designed?

⇾  Tests & benchmarks
How do the test suites look? What CI pipelines do they have, profiling? 

Don’t just use these libraries. 
Read them.

That will build your intuition and help you get better at your craft, either as a curious learner or an engineer as part of a larger team at your job.

---
I distill my learnings with my audience on The AI Merge.
Check it here (8700+ engineers already do that) https://buff.ly/2rwqlNL

I have a surprise for my European audience, for NVIDIA GTC 2026 (March 16–19)

🎁 I’m giving away 1 x NVIDIA DGX Spark to someone who joins me for Virtual NVIDIA GTC 2026. 

I’ve spent the last 6 months building on NVIDIA DGX Spark, and when building AI Systems locally, the Spark is an incredible machine to learn, prototype, fine-tune, build and iterate fast.

Especially if you want more control over your entire stack.

NVIDIA GTC is the premier global AI conference, where developers, and researchers come together to explore the next wave of AI across every industry and application.

✅️ Virtual GTC is free. 

𝗛𝗼𝘄 𝘁𝗼 𝗲𝗻𝘁𝗲𝗿 𝘁𝗵𝗲 𝗗𝗚𝗫 𝗦𝗽𝗮𝗿𝗸 𝗴𝗶𝘃𝗲𝗮𝘄𝗮𝘆:
𝟭. 𝗥𝗲𝗴𝗶𝘀𝘁𝗲𝗿 for GTC 2026 using my link: https://buff.ly/aKgT9BA
𝟮. 𝗔𝘁𝘁𝗲𝗻𝗱 at least 1 virtual session (keynote does not count) - virtual sessions are free
𝟯. 𝗙𝗶𝗹𝗹 𝗼𝘂𝘁 𝗺𝘆 𝗴𝗶𝘃𝗲𝗮𝘄𝗮𝘆 𝗳𝗼𝗿𝗺 𝗮𝗳𝘁𝗲𝗿 𝗮𝘁𝘁𝗲𝗻𝗱𝗶𝗻𝗴

I’ll share the giveaway form link as we get closer to GTC 2026. 𝗦𝘁𝗮𝘆 𝘁𝘂𝗻𝗲𝗱.

Here are a few sessions that caught my eye (links below):
1. AI Factories in Europe: Building the Foundations for Scalable Intelligence [S81899] 
2. Accelerate AI Through Open-Source Inference [S81902] 
3. Teach AI to Code in Every Language With NVIDIA NeMo [S82306] 
4. From Data to Meaning: Vision-Language Models Shaping the Cities of Tomorrow [S81867] 
5. Your Learning Pathway: Get Certified for Career Success [C81544] 

If you’re planning to attend, remember to register here: https://buff.ly/aKgT9BA 

Follow me here and stay tuned for the giveaway form.

Underrated advice to grow as an Engineer

1. Go Depth first (2–3 years)
Pick one stack (language + framework + system) and get good at it.

Breadth gets easy after.
I know it's tempting to know a bit out of everything, but you'll spread yourself too thin, too soon.

2. If you're a junior/mid - talk to senior/staff engineers (smartly)

Don't ask “What should I do?” 
Instead, ask: “Here’s what I tried, here’s what happened, what would you check next?”

They've seen enough systems and code to spot patterns early on.

2. Get good at Code Reviews
Express your points, think past the small code component - oversee the entire system.
Don't nitpick for code changes unless necessary. 

3. No Vibe-coding
With each feature/component you build, spend time gathering context, asking for options, plan, review, verify and write the hard paths yourself.

One practice that works for me is, after drafting a plan for my coding Agent (in Plan Mode), I ask if it's got any questions for me, before starting to work.

It usually comes up with great points.

3. Study outside work
That's an easy switch.
You do it already by consuming content, scrolling through posts and watching videos.
But make sure you focus on what actually helps you, not all the news/hype out there.

Work getting better, use AI as leverage.

A better way to manage complex configurations in your AI/ML projects.

AI/ML systems usually have more moving parts and more configuration surfaces (data, training, evaluation, inference, orchestration). Keeping track of which configuration subset was used can get difficult fast. 

Sure, as per Factor III (Config) in 12-factor apps, environment-specific config can be stored as env variables.

However, env vars alone are usually not enough. You might need hierarchical configs, runtime composition, overrides etc.

For instance, on a current project I'm working on, I have:
- 6 MLOps pipelines that build the raw_data > deployed model lineage.
- 2 edge services that ingest video, route frames to Inference Servers
- 3 MCP servers, each with their own connection details, tools and resources
- 2 Agents, wired through A2A - each with their own subset of instructions and MCP

In this setup, during testing, services often require runtime overrides (MLOps pipelines), composition or swap (training/eval), and sometimes they share the same configuration domain. 

When configuration gets more complex, I found Hydra + OmegaConf to be the way to go.

Here's what you need to know:
1. 𝗢𝗺𝗲𝗴𝗮𝗖𝗼𝗻𝗳 is a hierarchical configuration system explicitly designed for complex applications like ML pipelines. 

You can:
↳ Merge multiple configuration files directly through runtime composition.
↳ Access fields via attribute notation "config.model.optimizer" or dict notation config["model"]["optimizer"]
↳ Add Type safety against schemas such as dataclasses or Pydantic base models.

2. 𝗛𝘆𝗱𝗿𝗮 builds on top of OmegaConf, creating a full-featured framework.

You can:
↳ Group multiple YAMLs into a single configuration file and reference them directly by filename.
↳ Run a sweep, same application - multiple configurations in parallel.

Concrete examples:
1. Your hydra config could look like this (config.yaml)
``` config.yaml
defaults:
  - data: mydata
  - training: trainAB
  - inference: inferAB
  - _self_ 
```

When loading the `config.yaml` file using Hydra, we get a nested configuration object, from which we can access any field 
(cfg . inference . batch_size / cfg . training . batch_size)

2. Override any config value at runtime. 
For example, if in the config.yaml, if you have:
```
model:
 optimizer: SGD
```

At runtime, you can override the value with:

```
python train model.optimizer=Adam
```

𝗧𝗵𝗶𝘀 𝗮𝗹𝘀𝗼 𝘄𝗼𝗿𝗸𝘀 𝗳𝗼𝗿 𝗻𝗲𝘀𝘁𝗲𝗱 𝗰𝗼𝗻𝗳𝗶𝗴𝘀.
↳ Run multiple configurations simultaneously using Hydra multirun.

```
python train -m data.dataset=A1, B12, C55
```

This will launch multiple runs (one per dataset value) and create separate output directories for each run. 

---

I share more practical advice on AI Fundamentals, Engineering and Systems.
Join here (free): https://buff.ly/iqFpm9x

Qwen3.5 4B and 9B around the corner? 👀

Code is cheap nowadays. Design and planning is not. 

Some, keep measuring output by volume. 

I've been doing AI Assisted Programming for the past 2.5 years, on real codebases.

Online, I keep seeing many claim to push 10k Lines of Code a day to production and not reviewing anything.

I think that’s mostly marketing.
The more code added to a codebase, the larger the surface area.

Software isn't written once. It's designed to evolve.
What works now, should be easily changed or improved in the future.

The question you should ask: 
“Can we ship changes that are consistent and safe to release?”

A few safeguards that actually worked for me:
1. Treat AI code as proposal, not truth. Don’t accept diffs you can’t explain.
2. Put a harness on the agent. (Cursor Rules / AgentsMD / ClaudeMD / Skills / Plugins)
3. Keep Agent Rules, Skills and Plugins small, scoped and enforceable.
4. Add guardrails only after noticing Coding Agent failures.
5. Keep PRs reviewable. Minimal context + a 1-pager attached on the PR for big moves.
6. Reviews should't focus on preference. Skip nitpicks. Focus on the entire workflow.
7. Pre-commit + CI parity. Catch drift before you waste a CI cycle.
8. Write a handoff note to your tomorrow-self so you catch-up faster on the next day.

For an Engineer, AI helps, but only when you stay in control of constraints.

---
I explain complex topics in AI, AI Engineering and building AI Systems.

For free.

Join here: https://buff.ly/DnpHEXD

Six Tips on AI Assisted Programming

Head of AI/ML (10 years) with a strong background in Machine Learning and MLOps across various domains including information retrieval (Search, Recommendations, RAG-systems & ChatBots), NLP, and Customer Analytics. Brand new: AI Agents!

I specialize in guiding cross-functional teams of ML Engineers, Software Engineers, etc. to build scalable, event-driven ML solutions with a focus on robust MLOps practices using Python/PySpark and cloud platforms like AWS, GCP, etc. - (GCP Certified ML Engineer)

Currently, at HeyJobs, I have been fortunate to initiate and lead multiple complex ML projects (Search, RecSys, NLP and AI Agent) that that have significantly outperformed previous benchmarks and generated substantial business value. Additionally, I also designed and lead the implementation of a new MLOps and monitoring framework that has been embraced across all Data Science teams within the company.

Additionally, I love learning the latest advancements in the ML space, always exploring new ideas to push boundaries and improve outcomes. I’m also dedicated to mentoring and sharing knowledge, focused on building systems that drive innovation and business outcome.

Shantanu Ladhwe

I’ve hired many AI/ML engineers.
Two things consistently make the difference.
If you focus on just these, your odds jump. 👇

1️⃣ 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗼𝗻 𝘆𝗼𝘂𝗿 𝗖𝗩 
What I actually scan for:
→ A real link I can click (repo/app) - It's not necessary for senior positions
↳ A good README having an architecture sketch
(data → retrieval/inference → serving)
↳ Mention of numbers: p95 latency, cost per request, eval metrics, etc.
↳ Cloud infrastructure related tools used
↳ Aware of CI/CD; MLOps principles
↳ Postmortem note: what broke and how you fixed it (good to have)

🔹 𝗛𝗼𝘄 𝘁𝗼 𝗽𝗿𝗲𝘀𝗲𝗻𝘁 𝗶𝘁 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗖𝗩:
“Built Hybrid Search for X (small example)
→ p95 was around 700 ms,
→ cost $0.004/request, 
→ eval metric hit@10 was 0.83;
→ architecture was FastAPI + Postgres + OpenSearch + LangGraph;
→ Ops: Docker, CI, alerts;
→ Reduced latency by 42% with caching + batching”

2️⃣ 𝗔𝗜/𝗠𝗟 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝗲𝘀𝗶𝗴𝗻 (𝘁𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝘆𝗼𝘂 𝗰𝗮𝗻’𝘁 𝘀𝗸𝗶𝗽)
How to think out loud:
→ Start with the goal & SLOs (e.g: p95, quality, cost/user)
↳ Draw data flow first (very important)
(ingest → store → retrieve → infer → feedback)
↳ Call out trade-offs (latency vs quality, freshness vs cost)
↳ Add reliability: caching, fallbacks, and observability

🔹𝗣𝗿𝗼𝗺𝗽𝘁𝘀 𝘁𝗼 𝗿𝗲𝗵𝗲𝗮𝗿𝘀𝗲 (𝘁𝗶𝗺𝗲 𝘆𝗼𝘂𝗿𝘀𝗲𝗹𝗳):
✅ “Design a RAG for 1 Million PDFs; latency <1.5 sec. 
Where do caching and rerankers live?”
✅ “Deploy an LLM assistant; show routing 
(small→big model), cost guardrails, fail-open paths.”
✅ “Make it resilient to data drift; describe eval gates, rollbacks, shadow tests.”

🔹 𝗪𝗵𝗮𝘁 𝗜 𝗹𝗶𝘀𝘁𝗲𝗻 𝗳𝗼𝗿 𝗶𝗻 𝗴𝗿𝗲𝗮𝘁 𝗮𝗻𝘀𝘄𝗲𝗿𝘀:
→ You clarify ambiguity, then choose constraints
↳ You know where latency hides (I/O, tokenization, rerank, tool calls)
↳ You measure cost & quality per request (and have a plan to bend both)
↳ Retries, alerts, etc.

🔹 𝗔 𝘀𝗶𝗺𝗽𝗹𝗲 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝘁𝗼 𝗿𝗲𝗺𝗲𝗺𝗯𝗲𝗿 (𝘄𝗿𝗶𝘁𝗲 𝘁𝗵𝗶𝘀 𝗳𝗶𝗿𝘀𝘁):
1) Goal/SLOs → 
2) Data & storage → 
3) Retrieval/Model path → 
4) Latency/Cost plan →
5) Reliability (caching, queues, fallbacks) → 
6) Observability (traces, metrics, logs) →
7) Evals/Feedback → 
8) Risks & next steps

--
♻️ Repost if you found it helpful!
➕ We often this discuss on these topics here 👇
➕Join 40.000+ AI/ML builders here: https://lnkd.in/ds_SzEUH

Head of AI/ML - AI Agents, RAG, NLP, Recommenders, Search & MLOps

Many wouldn't like this:
If I’m hiring for an AI Engineer…
I’m not going to ask 5 RAG flows or 10 agent flows.

Here’s what I really want to know 👇

1. Can you design a system end-to-end? from ingestion to serving? what are the bottlenecks?
2. How would you estimate costs? how to reduce it?
3. How would you reduce latency? what is a good tradeoff of latency vs quality? 
4. Do you really need self-hosted LLMs? When is it needed? 
5. How would you fine-tune language models on user behavior? which framework? what about model serving?
6. How would you construct the dataset, what about loss function? what about MLOps? 
7. Which database would you use and why? Vector DB? SQL? NoSQL?
8. What metrics would you track? How? 
9. What about system monitoring? How would you debug failure cases?
10. What about feedback loop? how would we track and evaluate?
11. How would you make the system more deterministic?
12. How would you replace embedding models and backfill embeddings without any downtime?
13. What are the fallback mechanisms?

My favourite:
1️⃣ How would you solve this problem without LLMs or vector DBs?
2️⃣ Can you solve the problem with classical IR, rules, heuristics?
3️⃣ How would you make the system more deterministic?
4️⃣ Can you explain tokenization and embeddings from scratch?
5️⃣ What happens during fine-tuning? optimizer, learning rate, layer freezing? - not just “let's use LoRA”

These are just a few top-of-mind.

The reason:
I’ve seen engineers spend months building something flashy and complex that never made it to production.
I want engineers who understand the fundamentals, not just the hype.

I don’t want a flashy agent.
I want a resilient system.

NOTE:
Many of these questions are answered in the below self-paced course launched.

Do check it out here - https://lnkd.in/dtby5ZPU

(This is from real-world experienced professionals!)

--
♻️ Repost to share with others :)
🔔 Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ serious AI/ML builders here: https://lnkd.in/ds_SzEUH

Head of AI/ML | 143k+ audience | AI Agents, RAG, NLP, Recommenders, Search & MLOps

Most AI/ML book lists age fast. 
These don’t. 
As Head of AI/ML, these are my evergreen recommendations.👇

📌 Foundations:
1️⃣ 𝗙𝗹𝘂𝗲𝗻𝘁 𝗣𝘆𝘁𝗵𝗼𝗻 by Luciano
A deep dive into Python far beyond basic tutorials.
Perfect for writing modular, clean, scalable code the kind agents need. 
(could be skipped if online tutorials work for you)
🔗 Book link - https://lnkd.in/dD8ss6cK

2️⃣ 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝘀 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝘁𝗶𝘀𝘁𝘀 by 𝗣𝗲𝘁𝗲𝗿 & 𝗔𝗻𝗱𝗿𝗲𝘄 
A must-have foundation that stays useful across roles.
🔗 Book link - https://lnkd.in/dBWucpRX

3️⃣ 𝗣𝘆𝘁𝗵𝗼𝗻 𝗡𝗮𝘁𝘂𝗿𝗮𝗹 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗖𝗼𝗼𝗸𝗯𝗼𝗼𝗸 - Zhenya & Saurabh
A very foundation of NLP - Tokenization, n‑grams, stop‑words, stemming, TF‑IDF, BM25, NER. 
🔗 Book link - https://lnkd.in/d5aPp7ym

4️⃣ 𝗡𝗟𝗣 𝘄𝗶𝘁𝗵 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿𝘀 by Lewis, Leandro & Thomas Wolf
A solid foundation before diving into LLMs. Its highly important to know the pre-LLM era - comes very handy when working on Language data.
🔗 Book link - https://lnkd.in/diUjyCeg

📌 ML Systems
5️⃣ 𝗗𝗲𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 by Chip
The perfect start for understanding real-world ML development. Covers everything from data pipelines to model deployment.
🔗 Book link - https://lnkd.in/dY8NJMRk

6️⃣ 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺 𝗗𝗲𝘀𝗶𝗴𝗻 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 by Ali & Alex
A goldmine for structured thinking. Not just for interview preparation - it gives a good overview of how complex ML system could get, this definitely broadens your thoughts.
🔗 Book link - https://lnkd.in/dC4V5mbi

📌 LLM + AI Agents:
7️⃣ 𝗛𝗮𝗻𝗱𝘀-𝗢𝗻 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 by Jay & Maarten
From embeddings to RAG to fine-tuning - complete coverage
🔗 Book link - https://lnkd.in/ddb7kZ7D
🔗 Code link - https://lnkd.in/de52uRmQ

8️⃣ 𝗔𝗜 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 by Chip
Covers LLMs, RAG systems, infrastructure choices, multi-modal setupsm, very current, very practical.
🔗 Book link - https://lnkd.in/dqwDjHVa

📌 Advanced Systems:
9️⃣ 𝗗𝗲𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝗗𝗮𝘁𝗮-𝗜𝗻𝘁𝗲𝗻𝘀𝗶𝘃𝗲 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 by Martin 
I don’t see many mention this book but it is a
must-read to understand distributed systems, storage, and backend reliability. 
🔗 Book link - https://lnkd.in/dGG6VAaA 

--
♻️ Repost if you find it insightful :)
➕ Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ AI/ML builders here: https://lnkd.in/ds_SzEUH

FastAPI is free
PyTorch is free
Github is free 
Ollama is free
Docker is free
MLflow is free
Airflow is free
Jupyter is free 
DVC is free
Grafana is free
VSCode is free
Streamlit is free
PostgreSQL is free
Google Colab is free 
Hugging Face is free

Even the best LLMs, embedding models, and monitoring tools? Free.

𝗧𝗵𝗲 𝗮𝘄𝗸𝘄𝗮𝗿𝗱 𝘁𝗿𝘂𝘁𝗵:
Most "AI companies" charge thousands for what you can build with these free tools.

With a laptop and solid internet, you can build production-grade AI systems today!

Every tool you need to become a top 1% AI/MLOps engineer is already out there.

Every concept you need to master:
- monitoring,
- A/B testing,
- RAG pipelines,
- vector databases,
- model versioning,
- prompt engineering,
- experiment tracking,
- agent orchestration,
- automated ML pipelines - it's on GitHub, in the docs, in open courses, waiting for someone willing to break stuff and build again.

Nobody is stopping you from:
- reading research papers,
- joining the AI community,
- building your first RAG system,
- contributing to open-source ML projects,
- deploying your first ML model to production.

You can:
- create AI agents,
- automate ML pipelines,
- practice chunking strategies,
- deploy a local RAG on your laptop,
- build semantic search with free embeddings,
- set up model monitoring and experiment tracking
- and see exactly how modern AI applications work at scale behind the scenes.

Don't wait for:
- a PhD,
- expensive GPUs,
- the "perfect dataset",
- expensive cloud credits,
- enterprise infrastructure,
- someone's course or certification to validate your AI skills.

The models are there.
The community is there.
The frameworks are there.
The infrastructure patterns are there.

Start shipping systems.
What are you waiting for? 
Go build!

💭 Comment below your favourite free tool 👇 

--
♻️ Repost if you are AI/ML builder
➕ Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ AI/ML builders here: https://lnkd.in/ds_SzEUH

Stop using Streamlit for everything...❗
Here are 4 better alternatives 
(with real GitHub use cases) 👇

1️⃣ 𝗚𝗿𝗮𝗱𝗶𝗼 – 𝗙𝗼𝗿 𝗠𝗟 𝗱𝗲𝗺𝗼𝘀 & 𝗾𝘂𝗶𝗰𝗸 𝗶𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝗲𝘀
✅ Build UIs with just a few lines of Python
✅ Hugging Face integration + easy sharing
✅ gr.Blocks supports complex layouts
✅ Great for deploying models straight from notebooks

2️⃣ 𝗥𝗲𝗳𝗹𝗲𝘅 – 𝗙𝘂𝗹𝗹-𝘀𝘁𝗮𝗰𝗸 𝗮𝗽𝗽𝘀 𝗶𝗻 𝗽𝘂𝗿𝗲 𝗣𝘆𝘁𝗵𝗼𝗻
✅ No JS or React needed – just Python
✅ UI + backend logic unified
✅ Built-in auth, routing, deployment
✅ Perfect for tool-like AI apps

3️⃣ 𝗧𝗮𝗶𝗽𝘆 – 𝗣𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝗱𝗮𝘀𝗵𝗯𝗼𝗮𝗿𝗱𝘀 𝗳𝗼𝗿 𝗠𝗟 𝗮𝗽𝗽𝘀
✅ Great for simulations and “what-if” analysis
✅ Visual pipeline management
✅ Role-based access + scheduling support
✅ Mix UI + orchestration in one app

4️⃣ 𝗖𝗵𝗮𝗶𝗻𝗹𝗶𝘁 – 𝗕𝗲𝘀𝘁 𝗳𝗼𝗿 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 / 𝗖𝗵𝗮𝘁 𝗨𝗜𝘀
✅ Built-in support for handling chat inputs and agent steps
✅ Easy control over message flow and intermediate outputs
✅ Event-based structure for better agent orchestration
✅ Clean API to define conversational logic & tool execution

💬 Which one do you use? Have more to share?
Drop links and ideas below. 👇

🔗 I’ll add all the official links in the first comment.

---
♻ Repost to help others!
🔔 Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ AI/ML builders here: https://lnkd.in/ds_SzEUH

The AI/ML job market is brutal.
But not for the reason you think.

Two ML engineers interview for the same role.

Same background.
Same tech stack on the resume.
One gets hired. One gets rejected.

The difference was never about which tools they knew.
It was about how deeply they thought about the problems they solved.

This is exactly from my personal experience of hiring.
Confirmed by several other hiring managers I have spoken to.

Plenty of talent available. 
Very few who have the skills for the role.

𝗪𝗵𝗮𝘁 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗯𝘂𝗶𝗹𝗱𝘀 𝗱𝗲𝗽𝘁𝗵 𝗶𝗻 𝟮𝟬𝟮𝟲:

1. Solve a real problem around you.
Look at your job, your team, your daily friction.
Build something that actually reduces it.
Use AI to move faster, but own the design and the failures yourself.

2. Document every decision.
Why this architecture. What broke. What you would change.
That is your real portfolio. Not the GitHub repo. The thinking behind it.

3. Contribute to production-grade open source.
Ex: vLLM that teaches you how real inference handles memory and batching at scale. Try out LitServe, DeepEval, and ZenML are beginner-friendly and used in real systems. One merged PR signals more than ten cloned repos.

4. Or build something tiny yourself.
A pip-installable package that solves one specific friction.
A prompt version tracker. A token budget monitor. A retrieval quality logger.
Small scope. Real problem. Your design decisions end to end.

5. Be opinionated.
If you have debugged five retrieval pipelines with the same failure, say so.
If you have run evals across three LLM providers, share what you found.
Depth without a voice stays invisible.

6. Ask harder questions before you build.
Not "which model should I use?" but "does this model need to exist?"
Not "how do I improve accuracy?" but "what does wrong look like for this business?"
That is the switch from junior to senior thinking.

The engineers who thrive in 2026 will not be the ones who built the most.
They will be the ones who thought hardest about what they were building and why.

Remember:
Curiosity compounds. Tutorial certificates don't.

What do you think?

--
♻️ Repost to share with others :)
🔔 Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ serious AI/ML builders here: https://lnkd.in/ds_SzEUH

🚨 NVIDIA has a lot of AI courses.

These are the 8 I’d actually recommend for AI/ML engineers in 2026 👇

1️⃣ Generative AI Explained
→ Best for getting aligned on GenAI concepts without hype.
🔗 https://lnkd.in/d_-ZZewf

2️⃣ LLMs with RAG
→ Practical path to retrieval-grounded AI systems.
🔗 https://lnkd.in/gqdw6xNa

3️⃣ Building RAG Agents
→ Where RAG meets tool-driven agent workflows.
🔗 https://lnkd.in/gbuQNQ7G

4️⃣ Introduction to CUDA
→ Critical if you want to understand performance bottlenecks.
🔗 https://lnkd.in/gHtAWFMv

5️⃣ Building Video AI Applications
→ Useful for multimodal production use cases.
🔗 https://lnkd.in/dFjeiEGE

6️⃣ AI on Jetson Nano
→ Great for edge deployment thinking.
🔗 https://lnkd.in/d2pUhBT3

7️⃣ Digital Fingerprinting
→ Security/authenticity layer is increasingly important for AI content.
🔗 https://lnkd.in/gcxKvMc4

8️⃣ Machine Learning Systems (from your broader list)
→ Connects model work with production architecture discipline.
🔗 https://lnkd.in/dkiGZisg

If I had to pick only 3 for most engineers:
RAG + RAG Agents + CUDA basics.

---
♻️ Repost to help others!
🔔 Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ AI/ML builders here: https://lnkd.in/ds_SzEUH

This NVIDIA RAG blog teaches you
5 key configurations can directly improve enterprise RAG accuracy.
Most teams implement only 1–2.👇

If you’re building RAG with NVIDIA models, 
this is the architecture pattern worth understanding.

I went through the Enterprise RAG blueprint, 
and these are the parts that matter most for production teams:

1️⃣ Multimodal ingestion is the foundation.
Enterprise documents are not just plain text; they contain tables, charts, diagrams, scanned pages, and metadata. If your pipeline ingests only surrounding text, your agent will miss key evidence and produce incomplete answers.

2️⃣ Reasoning mode is the easiest quality lever.
For financial comparisons, multi-step logic, and quantitative queries, retrieval alone often fails. Reasoning helps the model interpret retrieved evidence and produce logically grounded responses instead of stitched snippets.

3️⃣ Query decomposition improves multi-hop reliability.
Many enterprise questions require evidence from multiple sections or documents. Breaking one complex query into smaller sub-queries, retrieving each piece, and then recombining results significantly improves answer completeness.

4️⃣ Metadata filtering improves both precision and speed.
Using business metadata (category, date, tags, compliance/security labels) narrows retrieval to the right document slice. This reduces noise, lowers latency, and improves contextual relevance, especially in large enterprise corpora.

5️⃣ Visual reasoning is essential for real-world enterprise data.
Important answers are often embedded in visual structures like charts, graphs, and infographics. Text-only pipelines miss these signals; multimodal reasoning makes those visual facts usable in final responses.

This is how modern NVIDIA-based RAG should look:

Ingest multimodal docs → 
embed/retrieve with metadata awareness → 
rerank → 
reasoning/decomposition when needed → 
optional VLM for visual-heavy queries → 
grounded generation with citations.

My take (as always):
RAG is NOT just chunk + embed + generate.
It’s now a data-and-retrieval architecture problem.

Teams that get these 5 layers right will ship agents that are:
more accurate, more reliable, and far more production-ready.

Hope this is useful 🙌 
(reference link in comments)

--
♻️ Repost if you find it insightful :)
➕ Follow me, Shantanu for production AI/ML/MLOps & careers
➕ Join 41.000+ AI/ML builders here: https://lnkd.in/ds_SzEUH

taplio

The Battle

Alex Razvant

Shantanu Ladhwe

Followers

Posts

Engagements

Likes

Performance ScoreAlex Razvant's LinkedIn Performance

Performance ScoreShantanu Ladhwe's LinkedIn Performance

Ready to grow your LinkedIn presence?

taplio

The Battle

Alex Razvant

Shantanu Ladhwe

Followers

Posts

Engagements

Likes

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Alex Razvant's LinkedIn Performance

Performance ScoreShantanu Ladhwe's LinkedIn Performance

Ready to grow your LinkedIn presence?

Performance ScoreAlex Razvant's LinkedIn Performance