Socure

- Owning ~40 Batch and Streaming pipelines and data models for Transaction, Vendor, Feedback datasets, ranging from few MBs to tens of TBs, serving 10s  of stakeholders company-wide.
- Redesigned the streaming solution to achieve a 1-minute SLA, processing 50K messages per minute, empowering teams for real-time fraud analysis.
- Actively contributing to company-wide cost reduction efforts, saving $25K monthly through infrastructure and pipeline optimization.
- Leading the team’s GovCloud (FedRAMP) projects, building secure and compliant pipelines to ensure data accessibility for stakeholders.
- Taking an active part in Tech Talks and company Tech Blog.

Skills: AWS (Kinesis, S3, Lambda, SFTP, DynamoDB), Kubernetes, Spark, Airflow, Scala, Python, SQL, Terraform, Snowflake, Delta, Databricks

Senior Data Engineer

Data Engineer

K Health

- Took ownership of existing techstack, revamped tools like Airflow, DBT by introducing coding standards and best practices to enhance efficiency and productivity of the team.
- Migrated from open-source Kafka to GCP PubSub, reducing hundreds of engineering hours spent on maintenance.
- Led data pipeline and modeling for the core event driven dataset, enabling access for teams across growth, marketing, finance, and more.
- Leveraged GCP Audit Logs to analyze usage and resource utilization, reducing BigQuery costs by 70% ($50K monthly).

Skills: GCP, Kafka, Beam, Python, Airflow, SQL, DBT, Terraform

Earnest Research

- Migrated ~10k lines of SQL codebase to Spark (Scala) for 5x faster and 2x cheaper processing.
- Developed data pipelines to process billions of transactional records, unlocking value for the company’s core product.
- Migrated the scheduler from Luigi to Airflow resulting in improved dev experience.
- Designed a new data lake structure for better and efficient consumption of data across teams.
- Led and delivered multi million custom projects successfully.

Skills: AWS (EMR, S3, Lambda, Redshift), Spark, Scala, Python, SQL, Luigi, Airflow, Snowflake

FNDtek

Founded the startup during college.

Worked on a variety of projects for multiple clients from web applications to data analytics.

Founder

Football News Guru

- Manage team of writers and technical staff to oversee day-to-day operations.
- Regular column on the latest football stories.
- Responsible for drafting content strategy for site and writers.
- Site promotion - including search engine and social media marketing.
- Ad Sales.

**Learned off the field strategies from Digital Marketing and Content Writing**

Avanza Solutions

Worked on a E-Banking Product "Ambit" as a full stack engineer.
Developed back end functionalities using Java (Spring MVC and Hibernate) while front end using Angular JS 2.0.

Software Engineer

Senior Engineer with extensive experience in leading, designing, and implementing large-scale distributed data systems and services, used by hundreds of internal customers.

Core Expertise:
- Batch and Streaming Pipelines
- Data Quality Frameworks
- Data Infrastructure
- Data Platforms

Core Skills:
Python, Scala, Spark, Beam, Hadoop, Amazon Web Services(S3, EC2, EMR, Athena, Redshift, CloudFormation, DynamoDB, DMS, Step Functions, RDS, Kinesis etc), Google Cloud Platform(Cloud Storage, DataFlow, Composer, BigQuery, etc), Luigi, Airflow, DBT, Kubernetes, Docker, Microservices, Terraform, Databricks, Delta.


Sharing knowledge at www.junaideffendi.com

Taplio

Junaid Effendi's Linkedin Analytics

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Junaid Effendi's Linkedin Analytics

Want to drive more opportunities from LinkedIn?

Junaid Effendi's Linkedin Analytics