Prophecy

Data Engineer

Expedia Group

Data Engineer - III

Amazon

▪ Salesforce to Redshift Ingestion - Migration from Informatica to Native AWS
     -> Tech Stack – Salesforce, Informatica, S3, Lambda, Glue, AppFlow, Redshift, SNS
     - Crafted generic scalable Native AWS solution for Salesforce to Redshift ingestion
     - It helped to move ingestion pipelines from third party tool Informatica and saved cost for heavy license fee
     - This generic framework helped other business units for smooth ingestion of newly onboarded Salesforce object into Redshift Datalake

▪ Incremental Ingestion pipeline – Employee Benefits Data
     -> Tech Stack – Shell Scripting, AWS CLI, S3, EMR, Glue, Redshift, SNS, QuickSight,PySpark
     - Build generic & optimized ingestion pipeline for highly critical & confidential Employee Benefits Data
     - Pipeline is designed in a way to handle GB’s of daily & weekly data together for different use cases like Audit, Payroll, Reimbursement, Education Reimbursement etc
     - Took complete ownership and worked closely with business teams to understand the requirements & deliver enriching dashboards

▪ Pipeline Optimization & Enhancement
     -> Tech Stack – Shell Scripting, AWS CLI, S3, EMR, Glue, Redshift, SNS, QuickSight, PySpark, Lambda
     - Enhanced & optimized multiple pipelines, built for different business units like Peoplesoft, Audit, AEM, Immigration, Accurate, MyDocs, Background Verification etc
     - Reduced execution time by 50% and improved alerting system for different edge cases
     - Leadership principals like Customer Obsession, Earn Trust and Think Big, helped me to keep improving existing systems

▪ Automated Alerting System for Job Monitoring
     -> Tech Stack – Python, AWS CLI, QuickSight
     - Created automated alerting system for Redshift load metrics and Job monitoring
     - It saved 1.5 Hours/day of manual efforts by each team member to monitor & prepare Daily Job Status

QuantumBlack, AI by McKinsey

▪ Feature Engineering For Telecom Client
     -> Tech Stack – PySpark, Kedro, Azure Cloud, Databricks
     - Created large scale & optimized pipelines for Telcom data using PySpark & Kedro framework
     - Worked closely with client in order to get business requirements
     - Implemented business logics to prepare clean & aggregated data for Customer Churn Analysis

Paytm

▪ Data Ingestion & Sync Process
     -> Tech Stack – Python, Hive, ElasticSearch, Scala Play Framework, SBT, EMR, Lambda, DynamoDB, Azkaban, Jenkins
     - Crafted data-sync logic by prioritizing datasets (High/Medium/Low tag) based upon criticality to meet SLO
     - Built premption logic to prioritize highly critical datasets when multiple low priority sync processes are running
     - Designed Rest API in data ingestion for retention of GA data in order to optimize cluster space
     - Added exception handling scenarios in data sync logic to fix multiple bugs
     - Fix for missing PG data from Kafka for UMP panel - Created a new pipeline to ingest missing data from HDFS to ElasticSearch
in case of cluster failure

▪ Near Real Time Data Pipeline – POC
     -> Tech Stack – Java, Spark, Kafka, Datastax Cassandra, Datastax studio, Zookeeper, Maven
     - Crafted a Cassandra based real time ingestion pipeline for marketplace data in order to help DWH team to reduce request
load from production MySQL. The Objective was to shift business users from production, to overcome data leaks & security
issues
      - Interacted with different business users to know about their use cases, ingestion tables, PII data and built data models
accordingly for faster insertion/updation of data
      - Setup web interface Datastax Studio for users to query real time data from Cassandra using LDAP authentication

▪ Hive Query Parser
     -> Tech Stack – Django, Python, NGINX
     - Query Validator and Optimization Engine - Created a Django web application to parse and validate user's hive queries. In
case of a bad query (missing partition columns/unbalanced joins), it also provides suggestions to improve the query - PII
detector – Built a Django web application to detect all running hive queries which are fetching PII data.

Software Engineer ( BigData & DWH )

OperaSolutions

▪ Procurement Spend Optimization (Pharmaceutical)
     - Developed CXO-level insights engine to manage USD 60Bn; engine enabled cost optimization using smart categorisation, benchmarking and anomaly detection
      - Crafted a Big Data based solution; organised structured & unstructured data
      - Built solution using Hadoop Ecosystem (HDFS, YARN), Spark and Python
      - Built a google translator API based solution to automate legacy translation engine; improved record aggregation accuracy by 50% and saved team 120 hours/month

Software Engineer - II ( BigData & Analytics )

▪ Trip Narrative Platform (Aviation)
     - Deployed an end to end solution for a leading US airlines; Aggregated a 360 view of customer's engagement throughout the life-cycle of the trip
     - Developed data pipelines from scratch; optimised data aggregation from 10+ independent sources and automated the ETL process to roll out the solution
     - The solution powers a web application; used by 1000+ CSRs and decision makers
     - Built application on RESTFUL API`s using Hadoop Ecosystem (HDFS, YARN), DataRush Applications (Distributed Processing Engine), SQL and Python

Software Engineer-1 ( BigData & Analytics )

Software Intern

GeeksforGeeks

▪ Data Structures & Algorithm Tutorials
     - Individually contributed 150+ articles on Data Structures (Arrays, Stack, Queue, Linked list, Tree) and Algorithms (Dynamic Programming, Optimised Searching)
     - Articles live on their website; received 30,000+ views and 2000+ upvotes
     - Published 3 YouTube videos on Data structures and Python; received 4000+ views

Technical Content Writer

Experienced Data Engineer with a demonstrated history of working in service and product companies. Solved data mysteries for different domains like Aviation, Pharmaceutical, FinTech, Telecom and Employee Services. Have designed scalable & optimized data pipelines to handle PetaBytes of data, with Batch & Real Time frequency. 

Got good exposure on different BigData frameworks (Hadoop, Spark, Hive, Sqoop, Flume, Flink, Kafka, Docker), Databases (MySQL, HBase, Cassandra, Redshift, Elastic Search), AWS Services (S3, Lambda, EMR, Glue, Cloudwatch, Redshift, SNS, SQS, Athena, Appflow), Dashboarding Tools (Grafana, Kibana, QuickSight, DataDog, Data Studio), Monitoring Tools (Airflow, Azkaban), Web Development (HTML, CSS, Scala Play, Django, Rest API, JavaScript, Ajax, JQuery), Good command over programming languages (Python, Java, Scala, Shell Scripting) and strong Data Structures & Algorithm fundamentals.

Big fan of taking ownership and working with Business Teams & Stake holders.

Content + Code = Future ❤️ I am in a good position where I can actively contribute to community and have been doing it since 2020. Creating amazing podcasts & data engineering related content on my Youtube channel (E-Learning Bridge).

LinkedIn has always helped me to grow in my career, best place for me to share my daily thoughts ❤️

Taplio

Shashank Mishra 🇮🇳's Linkedin Analytics

Shashank Mishra 🇮🇳

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Shashank Mishra 🇮🇳's Linkedin Analytics

Shashank Mishra 🇮🇳

Want to drive more opportunities from LinkedIn?

Shashank Mishra 🇮🇳's Linkedin Analytics