BARE International

Analytical Engineer

TTK Services Private Limited

Business Analyst

Resources to Learn Spark for free! 

✅ Spark Starter Kit on Udemy - Highly recommended for beginners. 

✅ Scala and Spark 2 - Getting Started Again on Udemy - 6 hours of extensive video material explaining the Spark 2 version based on the Scala version. 

✅ SparkByExamples - Such a 💎 , sometimes better than official documentation with some good examples. 
A website like: https://lnkd.in/gCH2U7ch

✅ Learning Spark - Second Edition by O'Reilly Corporation. 
An Extensive book, Skim through it to get a clear picture of the fundaments of Spark in the DataBricks environment. 
Link: https://lnkd.in/gziM96hH

✅ Finally the official documentation itself. TBH, it is quite technical but easy to understand most of the time. 
Link: https://lnkd.in/ggVXysGf

Hope this helps! 

#data #dataengineering #azure #spark #apachespark #aws #google

Gokul Arumugam

Who needs #aprilfoolsday when you spend hours designing a normalised database schema, only to have your client request a last minute change that turns it into a chaotic denormalized mess? 🙃

The Cluster Manager in spark like a Junior System architect. 😂 

Yes, that's right! 

It is responsible for the allocation of computing resources and the scheduling of tasks.

The roles and responsibilities are...

✅ Resource allocation: Cluster managers allocate resources, such as CPU and memory, to different applications and tasks running on the cluster.

✅ Task scheduling: Cluster managers schedule tasks to run on specific machines in the cluster based on the available resources and data locality.

✅ Fault tolerance: Cluster managers monitor the health of the cluster and take action to recover from failures, such as restarting failed tasks or reallocating resources.

Possible challenges with cluster managers are: 

⛔ Configuration complexity: Configuring a cluster manager can be complex and time-consuming, requiring a deep understanding of the underlying system and resources.

⛔ Scalability issues: As the size of the cluster grows, the complexity of managing resources and scheduling tasks increases, which can lead to scalability issues.

⛔ Performance bottlenecks: The cluster manager itself can become a bottleneck if it does not efficiently manage resources or schedule tasks.

Some of the most frequent issues that arise with cluster managers include:

🔴 Resource contention: If multiple applications are running on the same cluster, they may compete for resources, leading to performance issues.

🔴 Data locality issues: If data is not stored on the same machines as the tasks that need to process it, data transfer times can increase, leading to slower data processing.

🔴 Fault tolerance issues: Cluster managers may not be able to recover from certain types of failures, such as network errors or disk failures, which can lead to data loss or delays in data processing.

Hope this helps! Happy Learning! ❤️ 

#data #dataengineering #spark #apachespark #azure #aws

Do you know Execution memory and Storage memory can share their spaces when needed to accommodate the incoming request? 🤔 

Yes, that's possible! 

When a task is executed, Spark tries to minimize the amount of data that needs to be read from the disk and maximize the amount of data that can be kept in memory. 

✅ To achieve this, Spark uses a concept called "memory management" to determine how much memory should be allocated to Execution memory and how much to Storage memory.

✅ The memory management algorithm in Spark is based on a concept called "Unified Memory Management". This means that the same pool of memory is used for both Execution memory and Storage memory. When Execution memory needs more memory, it can take it from the Storage memory pool, and vice versa.

However, there are rules and policies in place to ensure that Execution memory always takes precedence over Storage memory, and that frequently accessed data is kept in Storage memory as much as possible.

Hope this helps! 

#data #dataengineering #azure #aws #spark #apachespark

What are the rules that dictate when and how storage sharing can occur between Execution memory and Storage Memory? 🤔 

✅ Execution memory always takes the upper hand over Storage memory. If Execution memory needs more memory, it can take it from the Storage memory pool.

✅ The storage memory can be evicted if Execution memory needs more memory. This means that if there is not enough memory available for Execution memory, Spark can reclaim some of the memory used by Storage memory.

✅ Spark uses an LRU (Least Recently Used) eviction policy for Storage memory. This means that the least recently used data in Storage memory is evicted first when there is not enough memory available.

✅ Spark allows users to control the amount of memory allocated to Execution memory and Storage memory using configuration parameters such as spark.executor.memory, spark.memory.fraction, and spark.memory.storageFraction.

To sum things up, storage sharing between Execution memory and Storage memory in Spark is based on a unified memory management algorithm that allows Execution memory to take memory from Storage memory if needed. 

Hope this helps! 

#data #dataengineering #spark #apachespark #aws #azure


⚡ Spark Worker Node Explained - Roles and responsibilities, How to handle failures, and Some challenges that arise when working with it.

Worker nodes are the machines in a Spark cluster that are responsible for executing the data processing tasks. Each worker node runs one or more Executors, which are responsible for processing the data. 

Their roles and responsibilities: 

✅ Receiving and executing tasks: From Spark driver and execute them using Executors.

✅ Managing resources: Requested by Executors, such as CPU, memory, and disk. They also manage data locality, ensuring that the Executors process data that is stored on the same machine or nearby machines.

✅ Reporting status: The worker nodes report their status, including how many resources they have available and the progress of the tasks they are executing.

✅ Handling failures: The worker nodes handle failures that occur on their machines, such as network errors, crashes, or out-of-memory errors. If an Executor fails, the worker node will restart it on the same machine or another machine in the cluster.

When a worker node fails, 2 most frequent challenges occur:

⛔ One challenge is that the data being processed by the failed Executor may need to be reprocessed, which can slow down the overall data processing. 

⛔ Another challenge is that the resources that were being used by the failed Executor may need to be reallocated to other Executors, which can be a complex and time-consuming process.

Happy learning! ❤️ 

Orchestration Tools: What They Are, How They've Evolved, and Why You Need Them?

✅ Orchestration tools are software applications that automate containerised applications' deployment, scaling, and management. 

They allow you to focus on writing code, while the orchestration tool covers the underlying infrastructure. 

⛳ Traditionally, orchestration tools were used to manage monolithic applications. These applications were typically deployed on a single server or cluster of servers. The orchestration tool would be responsible for starting and stopping the application and monitoring its health.

Now with the growth of microservices architecture, orchestration tools have become even more important. 

✅ Microservices are small, independent services that communicate with each other over a network. This makes it more difficult to manage microservices manually, as there are many more moving parts. 

✅ Orchestration tools can help automate the deployment, scaling, and management of microservices, making it easier to build and maintain complex applications.

Here are some of the most popular orchestration tools:

🎯 Kubernetes is an open-source orchestration tool that is widely used in the cloud-native community.
🎯 Amazon Elastic Container Service (ECS) is a managed orchestration service offered by Amazon Web Services (AWS).
🎯 Azure Kubernetes Service (AKS) is a managed orchestration service offered by Microsoft Azure.
🎯 Google Kubernetes Engine (GKE) is a managed orchestration service offered by Google Cloud Platform (GCP).

Hope this helps! 

#data #dataengineering #aws #azure #gcp #kubernetes

When you apply for the Job Vs. When you hear back from them 😂 

Relatable?

#data #dataengineering #azure #aws

The 2 types of important memory in Spark Executor! 

🎯 Execution Memory.
🎯 Storage Memory. 

Okay, what are these?

✅ Execution memory is the memory used by Spark to cache data during the execution of a task. This memory is used to store intermediate data, such as the results of a shuffle operation, and is freed after the task is completed. This memory is used for storing objects used during the execution of a task, such as intermediate data structures and temporary variables.

✅ Storage memory is the memory used by Spark to cache data that will be used across multiple tasks. This memory is used for caching data that will be reused in subsequent computations, such as the data that has been read from a disk or data that has been generated by a previous stage of computation. This memory is managed by Spark's caching layer, and the data is evicted from the cache when it is no longer needed.

Both execution memory and the storage memory are configurable in Spark, and their sizes can be adjusted based on the requirements of the Spark application. 

By default, execution memory and storage memory are allocated in a ratio of 60:40 respectively, but these values can be adjusted using Spark configuration settings.

Happy learning! ❤️

#data #dataengineering #azure #spark #apachespark

The Databricks community edition sucks, it is too slow to the point where it is not usable.

𝗪𝗵𝗮𝘁 𝗶𝘀 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 and why should you 𝗹𝗲𝗮𝗿𝗻 𝗶𝘁 𝗮𝘀 Data engineer? 😀 

❗Kubernetes, often referred to as K8s, is an open-source platform designed to automate deploying, scaling, and managing containerized applications.

It is a container orchestrator that performs the scheduling, running and recovery of your containerised applications in a horizontally scalable and self-healing way. 

❗Imagine you're working on a project with a team and you all have different types of computers and software. It could be a real headache to make sure your program runs correctly on everyone's machine. Containers solve this problem by including everything needed to run the program right inside the container.

Now, if you have lots of these containers, you need a way to manage them - that's where Kubernetes comes in. It groups containers that make up an application into logical units for easy management and discovery.

Why K8s?

✅ Service discovery and load balancing: Kubernetes can distribute network traffic to help balance the load between your application's components.

✅ Storage orchestration: Kubernetes allows you to automatically mount a storage system of your choice.

✅ Automated rollouts and rollbacks: Kubernetes can roll out changes to your application or its configuration while monitoring application health to ensure it doesn't kill all your instances at the same time.

✅ Automatic bin packing: Kubernetes automatically places containers based on their resource requirements and other constraints, while not sacrificing availability.

As a data engineer, learning Kubernetes because it can help you manage and scale your data processing tasks. 

For example, if you have a program that processes data and you want to run it on 1000 files, you could create 1000 containers, each processing one file. Kubernetes can help you manage this task, ensuring that the containers run smoothly and resources are used efficiently.

Some cons: 

❌ It has a steep learning curve. It's a complex system with a lot of different parts, so it can take some time to learn.

❌ It can be overkill for small projects. If you're just running a few containers, you might not need all the features Kubernetes offers.

❌ It's rapidly evolving, which means you have to keep up with the latest updates.

Share your thoughts in the comments!

#kubernetes #data #dataengineering #aws #azure #gcp

Taplio

Gokul Arumugam's Linkedin Analytics

Gokul Arumugam

Gokul Arumugam's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Gokul Arumugam's Linkedin Analytics

Gokul Arumugam

Gokul Arumugam's Best Posts .css-qqfgvy{font-size:var(--chakra-fontSizes-md);}(last 30 days)

Want to drive more opportunities from LinkedIn?

Gokul Arumugam's Linkedin Analytics

Gokul Arumugam's Best Posts (last 30 days)