Confidential

Senior Data Engineer

Kockpit Analytics

Business Intelligence Developer

Lushai Analytics

BI Analyst

Mizoram University

Junior Project Manager - Database and Visualization

Developer - Database and Visualization

Hi, I am a Data Engineer with experience in both the field of Data Analytics and Data Engineering.

➡️ Tech stack I worked so far : SQL, Power BI, Python, Apach Spark (Pyspark), Azure Cloud Service (ADF, ADLS-Gen2, Databricks, Azure Functionapp, Azure Webapp, etc), Snowflake Data Warehouse, Flask and Quart web development framework.

➡️ Worked on both on-prem as well as cloud technologies.

➡️ Experience with providing end-to-end data solutions started from building the ETL architecture to Power BI development and implementation as well as creating websites/applications on Flask and Quart framework.

➡️ Domains I worked so far: Retail, FMCG, Energy and Utilities and E-commerce.

➡️ Worked with clients from India, US and UK.

➡️ Closely worked with C-level management professionals for providing the best fit solutions by understanding the business requirements.

➡️ A team player and an individual contributor with experience of working in Agile methodology and Scrum framework.

"Bhaiya I want to become a Data Engineer but am confused about where to start, what to focus more and what to not. The internet shows a long list of tech stack - SQL, Python, Spark, Kafka, Azure, Snowflake, Airflow, AWS, dbt and the list goes on, which confuses me more"

This is one of the most common queries that I get from aspiring Data Engineers.

Here's something that one can follow:

➡️ Learn the fundamentals of data engineering such as Data warehousing, Data modelling, Basics of the ETL processes, etc.

➡️ Once done with learning the fundamental concepts, get good understanding of SQL and Python programming language.

These are some mandatory data engineering topics/skills that you can't skip at any cost.

➡️ Learn the fundamentals of distributed computing and get familiar with basic pyspark coding.

Even though organizations list n numbers of tech stacks in JD, these skills sufficient to get into the entry level Data Engineering role.

However, if you are already into any other data roles such as Data Analyst, Power BI Developer, etc., once done with learning the fundamentals, you can go through the architecture of your current project to find out the tech stack that is being used in the project and create the roadmap accordingly.

Bonus: While applying for the role, simultaneously you can pick any cloud and explore services that a data engineer uses on daily basis. This will act as an added advantage.

#dataengineer #data

Munna Das

Senior AI Engineer | Backend Engineer | Agentic AI | Generative AI

Joyeta Pal

SQL sub queries can be very confusing to a beginner.

As a beginner I also had a hard time understanding and writing sub queries.

Here's an approach that helped me a lot during my initial days with SQL :

➡️ Read the problem statement multiple time and note down what are the fields/insights the problem statement is demanding.

➡️ Do not focus on sub-queries initially but just treat the sub-query as an individual query and note down all the queries required to achieve the objective. You don't necessarily have to write it correctly but just for basic understanding of what you are trying to find.

➡️ Out of all the queries you wrote above, find out which one is the innermost query, the one which is independent of all the other queries and can be calculated directly from the dataset.

➡️ Write this independent query first, then find out that query which is directly dependent on this query and check on how you can integrate both these query together.

➡️ Likewise keep on identifying the next dependent query and keep on integrating.

Here's a simple example for better understanding:

📍Problem statement: Find the name of employees in a company whose salary is above the average salary of their development.

➡️ Here we need to find two things:
📍 employee who's salary is greater than average salary of his department
📍 average salary of the departments.


➡️ First query depends on the 2nd query and 2nd query is independent. So write down the 2nd query first :

📍 SELECT dept_id, avg(salary) AS avg_salary From employees GROUP BY dept_id

➡️ Understanding the logic behind the dependent query :

📍rough query : SELECT employee_name FROM employee WHERE salary > the answer that you are getting from the above query.

➡️ Understanding how to integrate both this query together to get the correct answer based on the employee's department and you are done ✅

P.S - Even if the person prefers CTE, they can use the above approach to understand how to create CTE and integrate it with the main query.

#sql #data

Working extensively on Azure helped me realize that though there are so many services azure offers, these are the services a Data Engineer will work with 80-90% of the time :

📍 ADF: Creating pipelines for the ETL process.
📍 ADLS: Data Lake for storing the data.
📍 Azure Function: serverless compute service for running event-driven code.
📍 Azure SQL DB: Relational Database.
📍 Azure Cosmos DB : NoSQL Database.
📍 Azure Synapse: Analytics and Data Warehousing.
📍 Azure DevOps: CI/CD and DevOps
📍 Azure Key Vault: Managing secrets and keys.
📍 Azure Databricks: Unified platform for data, analytics and Al.

Rest 10-20% may vary based on the project requirements.

#azure #dataengineer #sql #data

taplio

Munna Das's Linkedin Analytics

Munna Das

Performance ScoreMunna Das's LinkedIn Performance

What is Munna talking about?

Munna Das's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Munna Das's Linkedin Analytics

Munna Das

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Munna Das's LinkedIn Performance

What is Munna talking about?

Munna Das's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Munna Das's Linkedin Analytics

Performance ScoreMunna Das's LinkedIn Performance

Munna Das's Best Posts (last 30 days)