Quantela Inc.

Principal Engineering Manager I

- Setup and manage of Kubernetes with Istio gateway for on-prem production workloads.
- Capacity planning for production workloads (Both on-premise and cloud deployments). 
- Managing AWS and Azure cloud infrastructure.
- Owns deployment architecture of Quantela platform.
- Focus mainly on high availability, security and reliability.

Sr Engineering Manager​ 

Devops Tech Lead

LinkedIn

Linkedin Content Creator

Avika Networks

- Part of building Vayuna - an open-source alternative to virtualization technologies like VMware and Amazon AWS.
- Built Vayuna on top of KVM Virtualization with PHP (front-end), Perl & Shell Scripting (Backend), Mysql Database, Munin Graphs.
- Part of team building WAS (Wireless authentication server) to authenticate and authorize users to access the wireless access points. 
- WAS (Wireless authentication server) - Build on top of freeradius, PHP (front-end), Perl & Shell Scripting (Backend), Mysql Database.
- Part of building ISM (Internet Security Manager) for monitoring and blocking unwanted http requests that are going out and coming into a network.
- ISM is built with Perl scripting (CGI) and BASH Shell scripting.

Software Engineer

Hi my name is Mutha Nagavamsi 😊

On LinkedIn I talk about DevOps, SRE and Technology. Kubernetes mostly.

- 13+ years of overall experience.
- 4 years as a Software Developer + Ops.
- 6 years as technical lead in Devops and Cloud.
- 3+ years of experience in engineering manager role. 

🚀 What have I done so far?

• Knows Bash, Python, Golang.
• KVM Virtualization, Free radius.
• AWS, AZURE and ON-PREMISE environments.
• Managing Production Kubernetes + Istio cluster on-premise.
• Capacity planning. Engage effectively with stakeholders and collaborate.
• Troubleshooting Cloud, OS, network, and application level issues on Dev, QA & Prod.
• Contribute to technical documentation or educational content. Create & Execute OKRs.

I'm just a message or a comment away on my posts. Let's connect and talk 👍

🗓Last updated : 03-02-2024

If IPv6 has so many ip addresses when compared to IPv4 why don't we switch to IPv6? (a good question)

We can, but there's a problem. Not every device support IPv6. Because:

• Legacy systems don't support.
• Upgrading to IPv6 is expensive.
• The complexity to understand IPv6 is high. (significantly)
• Network address translation (NAT) with IPv4 still works. lol 

Cloud provider support:

AWS supports dual-stack in Amazon VPC
GCP supports dual-stack in VPC networks
Azure provides dual-stack support in Azure Virtual Network

dual-stack means to use both IPv4 and IPv6 addressing simultaneously.

So the IPv6 transition is happening.
Slowly but surely. Like a butterfly's beautiful transition.

Btw, do you write? 

If not, why not? For all the talent our land shares, deserves great writers (teachers ❤️)

Hope it's useful.
Appreciate a repost.
Follow along Mutha Nagavamsi. 
Thank you for reading my friend :)

Mutha Nagavamsi

We have 4.3 billion IPv4 addresses. That's huge.

(well, not really!)

Because we need IP addresses for more than 4.3 billion devices in 2024 and beyond.

Now, IPv6 on the other hand, has 340 undecillion addresses.

And my friend, 1 undecillion = 1,000,000,000,000,000,000,000,000,000 billion

If I say this number is monstrous, 
that would be a significant understatement. 

Wouldn't it? We are talking about 27 - 0's after 1.

Here's something you should also know.
IPv6 and IPv4 work differently. 

IPv4 uses Network Address Translation (NAT) to allow a single public IPv4 address on behalf on many private IP's inside the local network. 

IPv6 does not use NAT. Which means IPv6 don’t get that many-to-one leverage like with IPv4. 

And honestly, it doesn't need NAT. Just because of the scale of IP's available. 

But who knows, anything can change any time. For change is the law of life ❤️

Hope it's useful.
Appreciate a repost.
Follow along Mutha Nagavamsi. 
Thank you for reading my friend :)

In kubernetes, you can add a delay to pod termination to allow clean exit.

This is quite handy to release database connections to the app. how is this done? by using pre-stop hook (one good way!)

A pre-stop hook is a command that runs before the container is terminated. Here's how to use it in pod spec

spec:
 containers:
 - name: my-container
  image: my-image
  lifecycle:
   preStop:
    exec:
     command: ["sleep", "15"] 

The above prestop hook delays pod terminations by 15 seconds. 

And my friend, delaying is not always a step back, but a setup for a graceful shutdown ❤️

Hope it's useful.
Appreciate a repost.
Follow along Mutha for more.
Thank you so much for reading :)

Not many people know this but knowing about pps in AWS ec2 could save your application from sudden slowness.

I just read this. And I'm sharing with you. AWS EC2 can have packets per second (pps) limit. 

It is separate from overall bandwidth allowed on an EC2 instance.

Even if your EC2 instance is using overall bandwidth with less than the specified threshold, pps limit can be crossed because of small packet size. This can slow down network performance.

On a side note, a NAT gateway can process 1,000,000 packets per second. And it can scale too, for up to 10,000,000 packets per second. Beyond this limit you will see performance degradation. 

So, by splitting EC2 resources into multiple subnets, we can prevent any problems linked to pps.

And the lesson: Problems teach, if we are willing to learn ❤️

Hope it's useful.
Appreciate a repost.
Follow along Mutha for more. 
Thank you so much for reading :)

Do you know this? kubelet sends heartbeat to the api server every 10 seconds.

Here's are more interesting numbers.

API server's --max-requests-inflight (default:400) setting helps you set max concurrent requests api server can handle.

A Pod's terminationGracePeriodSeconds default is 30 (pod needs to shutdown in 30 sec or is forcefully terminated)

kube-proxy's --iptables-sync-period (default: 30 sec) is how often iptables rules are refreshed.

CoreDNS default TTL (Time To Live) for DNS records is 30 seconds.

That's all for today. Hope you learned something today. Appreciate a repost.

Stay true to yourself → And share stories (your stories) to inspire ❤️

Always have a hands-on time in the tech you are learning.

• Create CICD pipeline.
• Launch VMs on your AWS.
• Create your own docker image.
• Spin up your own Kubernetes cluster.
• And always automate daily tasks to save time. 

Because every minute you save, you add it to your life ❤️ 

Hope it's useful, a repost helps everyone.
Appreciate a follow Mutha Nagavamsi for no-fluff content.

What happens if you stretch your Kubernetes worker nodes beyond 3000, 4000 or 5000 nodes?

Well, if you are using default configs it's very hard to even get there. 
But even if you manage to do so, apiserver gets overloaded. 
And eventually control plane will slow down or even crash.

Similarly, you have 110 pod limit per node. What happens if you stretch beyond that point in a 5000 node cluster?

In that case, first, your Kubelet and your container runtime gets over loaded. 
Then eventually that load will come to api-server as well.

Just to be honest, you will see many more problems even before you reach the limits (5000 nodes + 110 pods/node).

But my friend, honestly acknowledging the challenges is the first step to overcome them ❤️

What do you think? Share your experience with scale.

Hope it's useful. A kind repost helps.

Lifecycle of a Kubernetes API request (via API server)

• Send API request (from user or controller)
• Authentication & Authorization (permission)
• Mutating Admission (ex: add new-labels)
• Object schema validation (by API server)
• Validating Admission (final checks no modification)
• Persisted to etcd (storing the data)

What happens after that?

Normally, a controller picks up and takes some action to get to the desired state.

For example: If a new Pod is created, the scheduler controller takes action and will assign it to a node.

And talking about action, consistent actions build confidence. And confidence, my friend, leads to greatness ❤️

Appreciate a repost.
Follow along Mutha to support.
Thank you so much for reading this :)
#kubernetes #devops

Do you know? Only 1 hour of kubernetes events data is stored in etcd with default config. 

Plus searching for events is hard sometimes. 

Is this enough for most small / medium scale clusters? Maybe!

But I think for larger clusters you may need better k8s events visibility. 
And so it's good to export these events to an external database.

How exactly?

Well, you can use kubernetes-event-exporter and ship data to sources like opensearch/elasticsearch and many others for visualization needs. Learned it few days ago, so wanted to share. 

Have you done it? Share your experience. Thank you for reading.

In Kubernetes, if you apply default-deny-egress network policy, where is the packet dropped exactly?

Here's the default-deny-egress from official doc.

kind: NetworkPolicy
metadata:
 name: default-deny-egress
spec:
 podSelector: {}
 policyTypes:
 - Egress

First things first, the packet is dropped by CNI plugin. Where it's dropped depends on network plugin you select.

For example: In Calico, rules are enforced by iptables. So that's where the packets would be dropped.

In Cilium, the packets are dropped by eBPF program in the Kernel. Not the iptables.

Hope that's a useful learning. Consistent learning is life's greatest adventure ❤️ 

Appreciate a repost.
Follow along Mutha Nagavamsi. 
Thank you for reading my friend :)

What happens when you build a new Docker image?

One or more image layers are updated based on your input changes in Dockerfile.

For context: Docker images are built layer by layer, each layer represents a specific instruction in the Dockerfile.

Here's how it works:

Docker compares the new Dockerfile instructions to the previous image layers.
Only the changed parts of the image are rebuilt and stored as a new image layer.
The updated Docker image consists of the unchanged original layers combined with the new delta layer.

The layered approach optimizes data transfer and build times.
That means faster container creation and deployment (in Kubernetes cluster). Saving us lot of time.

And time saved is → time gained my friend ❤️

Appreciate a sub: mutha.substack.com
A repost helps everyone.

I'm always fascinated to find how Kubernetes works.

Especially how the control loop works.

A fundamental K8s concept everyone should know.

It's simple. And it's all about making sure the desired state of your cluster matches the actual state.

Let's talk with an example. 

Assume you have your web server on a K8s cluster.

The desired state of app:
• 3 replicas of a web server pod

And lets say the actual state is:
• 2 web server pods running

The control loop notices the difference.

• desired state (3 replicas)
• actual state (2 replicas)

So it creates 1 new web server pod to match the desired state of 3 replicas. 

This is the concept that makes self-healing possible.

In this case, self-healing is when your cluster heals itself,
by creating one new pod. 
A new pod gets a life.

And just like that creation sparks life. And life creates love ❤️

Hope it's useful.
Appreciate a sub: mutha.substack.com
And Thank you for reading my friend :)

You can see the Kubernetes API request sent to the API server with a simple kubectl option.

You can also simulate a command without modifying anything. Just combine this with the --dry-run flag. Here's an example:

kubectl apply --dry-run=client -v=8 -f your-deployment.yaml

It will show you a detailed output, along with API request sent to the API server. It's quite useful while debugging issues. 

And the only way to solve any issue is to face it 💪

Share any other handy tips you know! For the community.

Btw, I'm working on a new video. This one will be hand-on. 

I will publish it here first https://mutha.substack.com (appreciate your support!)

Hope it's useful.
Follow along Mutha for more.
Thank you so much for reading :)

You probably know them, and many of you must be following them already.

If you don't know them or not following them, you are missing out.

Creators in devops/cloud area.
Listing people from my list (random order)
Based on content quality and longevity of work.

Syed Nadeem
Suman Chakraborty
Raghav Dua
Piyush sachdeva
Abhishek Veeramalla
Sushant Kapare
Sandip Das
Nana Janashia
Priyanka Vergadia
Mumshad Mannambeth
Govardhan (can't tag!)

Share your fav creator. Creator who educates, empowers and inspires ❤️

some people took this way too seriously. A weekend post my friend. BTW, I have 2 questions.

what are your plans for this weekend? Any good reading recommendations?

A good book.
A good video.
A good blog post. 

Anything that's latest and you enjoyed learning, do share. Thank you 🙏

#kubernetes #devops

What happens when Kubernetes pod is throttled?

(question that's asked quite a lot)

• Simple answer: your app gets slowed down. 
• Eventual evidences: Increased response times and errors.

Throttling is not totally a bad thing. It is a way to prevent node level resource starvation for other pods.

And here are 3 ways to prevent throttling scenario.

1. Load test. Poorly designed pod resource configs can cause issues.
2. Monitor for resource usage and adjust requests/limits.
3. Constantly optimize app performance.

Because perfection is not the goal, but progress is ❤️

Hope it's useful.
Appreciate a repost.
Follow along Mutha Nagavamsi 
Thank you for reading my friend :)

Kubernetes upgrade. Most people fear it. For the new and unexpected issues it brings.

But upgrades must be done. To get improved features, bug fixes and security enhancements. 

And the question now is, after the upgrade, what are the most important things one should check? Here's my take on the topic.

1. App functionality
2. App performance
3. Default config changes
4. Storage driver compatibility
5. Verify monitoring and alert triggers
6. Check backup and restore procedures

These are the minimum checks that must be done. What do you think? Am I missing anything? Let me know.

Plan and execute the plan. For planning is wisdom in action.

Hope it's useful.
Follow along Mutha for more.
Thank you so much for reading :)

Kubernetes has some magnificent defensive mechanisms.

And here's an important one.

Important because it can optimize API server performance.
Important because it keeps api-server alive.

How? By limiting concurrent requests coming to the API server.

How so? By setting "--max-requests-inflight" (Default: 400) in your api-server config. How cool is that?

Plus, you can also set "--max-mutating-requests-inflight" (Default: 200) to optimize concurrent mutating requests.

Mutating requests are about creating, updating, or deleting objects. And it's a good idea to optimize this during peak load.

Hope it's useful. That's all for today. For today offers a new day to learn and grow. Make full use of it ❤️

Appreciate a repost.
Follow along for more - Mutha

PS: Kubernetes has other defensive mechanisms, this is one among them. Thank you for reading.

OpenTelemetry is a powerful tool.

It is about collecting data and distributing it to different destinations.

What data and where?

• Traces to a tracing system like Jaeger or Zipkin
• Metrics to a monitoring system like Prometheus or Grafana
• Logs to a logging platform like Splunk or ELK/Opensearch Stack.

This flexibility enables you to use the best tools for different purposes. 
Allowing us to adapt to change (with new tools).

For the measure of intelligence is the ability to change, said Einstein ❤️ 

Hope it's useful.
Appreciate a repost.
Follow along Mutha Nagavamsi. 
Thank you so much for reading :)

Your k8s cluster can experience churn. If ...

(What is churn? - high rate of pod creation, deletion)

It's the first major problem I faced during my early days.

With Kubernetes, some churn is expected. 
Because K8s is a dynamic environment. 
Pods get created and deleted.  

But excessive churn will not do much good.

Why is churn a problem?

Because it increases cluster resource utilization. 
Because of frequent pod creation and deletion.

And the result? Performance degradation. Even in your production.

The fix?

Have descent resources for your workload.
Load test to come up with optimal pod resource requirements.
And always monitor, monitor and monitor. It helps you optimize things.

For optimization is like a compass in a stormy sea. It guides us all back to safety (our home) in tough times. Including in production.

And talking about production, I created a video about how you can troubleshoot application latency issue in K8s.

Please check it out here - youtu.be/tn3u-ZR9BwQ

Hope it's useful.
A repost helps big time ♻️
Thank you for reading my friend :)

Taplio

Mutha Nagavamsi's Linkedin Analytics

What is Mutha talking about?

's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Mutha Nagavamsi's Linkedin Analytics

What is Mutha talking about?

's Best Posts .css-qqfgvy{font-size:var(--chakra-fontSizes-md);}(last 30 days)

Want to drive more opportunities from LinkedIn?

Mutha Nagavamsi's Linkedin Analytics

's Best Posts (last 30 days)