DataExpert.io

Founder

Ardent AI

I think AI will disrupt data engineering in a significant way. I’m investing in Ardent to accelerate that disruption!

Angel Investor

Peony Health

I invested in Peony Health because I believe habits are greatly impacted by neurochemistry and Susie Chen is a technical founder with business chops who will deliver on their mission

Illuminate AI

I'm one of the new Bay Area chapter leads for Illuminate AI! 

I help make events better and build this community to the next level!

Chapter Lead

Mage

I'm advising Mage on how to make a data engineering friendly Airflow competitor that will be the next generation of job orchestration!

Advisor

A House in Utah

Former Child

SylphAI

I’ve invested in SylphAI because I’m passionate about building community in AI that helps everyone meet their growth goals and fulfill their potential

Airbnb

Changed Airbnb's availability definition from "did a host set a rule that blocked this night" to "can a trip be booked that contains this night" 

Consolidate the availability definition across the company. Saving >500 TBs in the cloud and shrinking the data set to 3% its old size.


Decreased the landing time of Airbnb's unit economics dataset from 3 days to 1 1/2 days. Built a long-term roadmap to further increase the quality of this critical dataset. 

Improved the quality of online systems at Airbnb to be easily compilable by Spark pipelines to compute metrics offline. 

Lead a team of 6 engineers to deliver on the MIDAS initiative to increase the data quality of the Commercial Products org. 

Upleveled smart pricing at Airbnb by improving the feature engineering and latency of the data used to train the smart pricing model.

Staff Data Engineer - TC $600k ($220k Salary $320k Stock $60k bonus)

Netflix

Built a machine learning feedback system that allowed security engineers to label corporate user behavior as risky or not risky.

Built Asset Inventory - a graph database solution that is a map of all of Netflix's cloud infrastructure.

Senior Software Engineer - TC ~$575k ($550k salary + 5% options)

I built a pipeline that measures the cloud infrastructure impact on AB tests, saving Netflix millions by allowing them to make smarter AB test rollout decisions.

Senior Data Engineer - TC ~$365k ($365k salary + 5% options)

Facebook

- Managed a 10 PB+ Hive data warehouse
- Consolidated and conformed company-wide growth metrics (across WhatsApp, Instagram, Messenger, and Facebook) into a single, company-wide view. 
- Optimized machine learning feature set generation pipelines (200+ TB/day) from having a 4 day latency to having a 1 day latency. While also dropping compute costs for those pipelines 4x. 
- Reduced core notification data set latencies from 36 hours to < 8 hours. 
- Migrated 50% of notifications pipelines from using Hive to use Spark, Presto, or real-time streaming.
- Cut compute cost from notifications pipelines by 40% over the course of 9 months.

Data Engineer - TC $185k ($135k salary, $50k stock)

I love to teach. I post about #dataengineering and #mentalhealth daily. 

Many years of data engineering experience doing growth analytics at Facebook, security infrastructure at Netflix, and pricing and availability infrastructure at Airbnb. 

You can find me by looking up EcZachly on all social networks 

Join my Data Learning Platform https://www.dataexpert.io

I’m happy to share that I’m starting a new position as Strategic Advisor at omi!

I believe in Nik Shevchenko and I want to start having a much more data rich life!

Zach Wilson

Founder @ DataExpert.io, use code BF for 40% off!

The most impressive model is Gemini right?

Data Engineering ​> AI Engineering and I’m tired of pretending it isn’t

A mistake many data engineers make is thinking real-time = streaming pipelines! 

Whenever a stakeholder says they want data in real time, you shouldn't default to dreams of Flink, Kafka, and watermarks. 

You should clarify with precision what an acceptable amount of latency is for this use case. 

Many times when a stakeholder asks for real-time data, it can be solved with an hourly batch pipeline. The incremental benefit to jump from hourly batch to streaming isn't worth it because it impacts the homogeneity of your suite of pipelines and makes the overhead maintenance much higher! 

Sometimes stakeholders say real-time and what they mean is "predictable refresh rates." This is a sign you need to do better as a data engineer at setting SLAs for your pipelines about when they'll refresh.

Hitting 500k followers here on LinkedIn feels like a dream come true!

I started posting every day back on December 12th, 2020 and I had no idea it would impact my life in such a profound way

I met so many amazing creators along this journey like Carly, Aishwarya, Joe, Chip, Delia, Li, Annie, Jhillika and many more! 

To celebrate 500k followers, You can use code 500k at DataExpert.io/500k to get 50% off! This deal ends today at midnight pacific time! 

How long until I hit 1 million?

SQL is deceptively complex. The order which things apply isn’t that intuitive and can be frustrating when debugging queries. 

Let’s talk about the ordering of a query and when each step is executed. 
Here’s the query well deconstruct. 

SELECT 
 city, 
 SUM(weed_smoked) as total_weed_smoked 
FROM weed_consumption 
WHERE is_legal = TRUE 
GROUP BY city 
HAVING SUM(weed_smoked) > 420
ORDER BY total_weed_smoked DESC 
LIMIT 69

First, SQL executes FROM and WHERE. So we take the weed_consumption table and filter to only rows where is_legal is True. 

Then we GROUP BY city to start our aggregation. 

Then we filter the aggregation to only cities that have smoked at least 420 pound of weed. HAVING only ever applies to the aggregated result. 

Then we apply SELECT and aliases.

After that we ORDER BY so we see the most Snoop-like cities first. 

Then we LIMIT to the top 69 cities.

Learning patterns and tricks can really elevate your  data engineering! 

- the sorted-merge-bucket join (SMB join) 
This type of join requires both the left and right tables to be sorted and bucketed on the join key. If they are, this join can happen without shuffling and is extremely fast. I used this technique at Facebook to save tens of thousands of CPU days. 

- the datelist data structure 
Sometimes having a partial history in the same row can dramatically increase performance. At Facebook, I used this concept to store the last 30 days of someone’s activity as an integer. 01010111… the first place is their activity today. The last place is their activity 30 days ago. This reduced representation can have a huge impact on performance.  

- write-audit-publish pattern 
This one is critical for data quality. This pattern treats publishing to production as a contract. Write to a staging table, run your quality checks, if they pass, move the data from staging to production. 

- idempotent ETLs 
Writing ETLs that generate the same data regardless of if you run them today or next week is very useful. Avoid current time stamps, unbounded date ranges, and non-parameterized filtering. Always have a “logical” date that filters the data sets you’re processing. Following this pattern makes backfilling much easier.

Hacking the Data Structures and Algorithms Interview

Join this live to understand how to master the data structures and algorithms round

We're making some important updates to the Snowflake + dbt boot camp on January 6th! 

- We are limiting seats to 60 students to keep quality of instruction high (we are already halfway booked up)
- We are adding Cortex lessons for both vector search and document parsing
- We are adding a lesson change data capture 
- We are adding an Iceberg and Snowflake module for those interested in the open data format! 

If you're interested in some early Black Friday deals on this boot camp, send me a DM!

The data engineer handbook is the 565th most starred GitHub repo of all time out of the 28 million public repository. Meaning it’s in the top 0.002% of all repos on GitHub! 

Thank you all for taking it to new heights!

We start the SQL module of the 50 day challenge today! We’ll be covering everything from analytical patterns, to window functions, and more. 

There are a mix of multiple choice and coding challenges this week!

DataExpert doesn’t offer a job guarantee because if we did, we’d have to 7-12x our prices which would make them much less accessible. 

You know who else doesn’t offer a job guarantee? Stanford, Harvard, etc. Their new graduates are experiencing the highest unemployment rate in their history! 

So which would you prefer, a $1800 boot camp that gives you a job referral network with no guarantee or a $18,000 job placement program with a job guarantee? 

Given the placement/promotion rate after 12 months of DataExpert program is ~30%, you’d have to raise prices so the 30% pay for the refunds of the 70%. 

This model incentivizes students to be lazy in their job search to get their money back and it’s why Harvard doesn’t offer a “job guarantee” on their education paths.

I'm launching a free data science interview boot camp with Vaishali Macwan tomorrow!

For the next four weekends, Vaishali will go over the most important things to land a product data scientist role!

- Week 1 Getting Noticed
You can't land a job without landing an interview. Master the job search skills quickly!

- Week 2: The fundamentals and interview formats
SQL is absolutely essential for data scientists. We'll be covering the different interview formats this week too

- Week 3: How to pass the product sense and company fit rounds
Product sense is one of the most difficult interview rounds for data scientists. Vaishali has mastered this round and will pull back the curtain!

- Week 4: The behavioral round and negotiation
These two rounds are critical to sealing the deal for your new data science job and getting paid the most you can!

You can join the free boot camp here: https://lnkd.in/gj7xxGU3

Anirudh is crushing the free 50-day challenge, you can still join here: https://lnkd.in/gzn3JwwU

Data & AI Engineer | AWS & Google Cloud Certified | ML & Data Pipeline Specialist | Data Engineering Content Creator (Substack)

SQL performance tuning is all about context! 🚀

I made a classic mistake on Day 9 of Zach's 50-Day Data Challenge. The scenario exposed a critical blind spot in my understanding of how partitioning interacts with indexing.

Here was the challenge: We have a large sales table partitioned by sale_date. The following query was running slowly:

SELECT region_id, SUM(amount)
FROM sales
WHERE sale_date >= NOW() - INTERVAL '90 days'
GROUP BY region_id

The options for the best index were:
• idx1: (sale_date)
• idx2: (region_id)
• idx3: (region_id, sale_date)
• idx4: (sale_date, region_id)

I picked idx4: (sale_date, region_id).
𝗜 𝘄𝗮𝘀 𝘄𝗿𝗼𝗻𝗴. The correct answer is idx3: (region_id, sale_date).

𝗠𝘆 𝗙𝗹𝗮𝘄𝗲𝗱 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴:
I assumed the filter column (sale_date) should always be first in the index definition, and the GROUP BY column (region_id) second. Classic isolation thinking.

𝗪𝗵𝘆 𝗶𝗱𝘅𝟯 (𝗿𝗲𝗴𝗶𝗼𝗻_𝗶𝗱, 𝘀𝗮𝗹𝗲_𝗱𝗮𝘁𝗲) 𝗶𝘀 𝗖𝗼𝗿𝗿𝗲𝗰𝘁:

Here is the key insight I missed:

𝟭) 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻 𝗣𝗿𝘂𝗻𝗶𝗻𝗴 𝗛𝗮𝗻𝗱𝗹𝗲𝘀 𝘁𝗵𝗲 𝗗𝗮𝘁𝗲 𝗙𝗶𝗹𝘁𝗲𝗿
Since the table is partitioned by sale_date, the database engine uses partition pruning to instantly eliminate irrelevant data partitions. The date filter isn't the primary bottleneck; the 𝘨𝘳𝘰𝘶𝘱𝘪𝘯𝘨 is.

𝟮) 𝗜𝗻𝗱𝗲𝘅 𝗢𝗿𝗱𝗲𝗿 𝗗𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗲𝘀 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗦𝗼𝗿𝘁
This is crucial:
 • idx4 (sale_date, region_id): The data is sorted by date first, resulting in mixed regions: [Region A, B, C, A, B...]. The database must perform an expensive in-memory sort or hash aggregate to group by region_id.

 • idx3 (region_id, sale_date): Data is sorted by region first: [All Region A rows together, all Region B rows together...]. The database can perform a highly efficient Stream Aggregate, grouping on the fly as it reads the pre-sorted index - requiring zero additional sorting.

𝟯) 𝗧𝗵𝗲 "𝗟𝗼𝗼𝘀𝗲 𝗜𝗻𝗱𝗲𝘅 𝗦𝗰𝗮𝗻" 𝗔𝗱𝘃𝗮𝗻𝘁𝗮𝗴𝗲
For partitions that only partially fall within the 90-day window, idx3 allows the database to use a technique called a Loose Index Scan (or Index Skip Scan): jump to the start of each region_id, scan the necessary date range within that region, and move to the next. Highly efficient when the leading column (regions) has low cardinality.

𝗥𝗲𝗮𝗹-𝗪𝗼𝗿𝗹𝗱 𝗜𝗺𝗽𝗮𝗰𝘁:
On a table with millions of rows and 50 regions, idx4 might force a massive, slow memory sort. idx3 streams results with near-zero memory overhead. That's the difference between a query that scales and one that doesn't.

Getting questions wrong is where the real learning happens. 💡

Day 9/50 ✅

#DataEngineering #50DayDataChallenge #SQL #DatabaseOptimization #Indexing

Being independent has made the sales cycle of my business awful. I could move my content towards making it easier for others to get their pieces of paper from another bigger platform. 

What do you think?

We grade full data models in the https://lnkd.in/gzn3JwwU !!

Data & AI @ NTT DATA | Ex - Snowflake, TCS | Databricks Certified Data Engineer Professional | SnowPro Core Certified | Python | SQL | PySpark | Databricks | Snowflake | DBT | AWS | Airflow | GenAI | Git

#5 Completed Data Modelling: Week 1 all quizzes, so today's challenge was little difficult as it was not a straight MCQ but rather a descriptive question where we had to design the data model with it's advantages and trade-offs and why and what we are doing!

Thank You Zach Wilson

The code is working again here: DataExpert.io/BF

AI is changing data engineering in dramatic ways but not replacing it! 

Starting January 6th, I'll be kicking off a 15-week boot camp that covers everything you need to know to become an AI-enabled data engineer who is skilled with AWS, Databricks, Snowflake, and AI Agents.

I'm currently raising money from YC and other investors to build a company where I empower other content creators to follow their dreams so this will be the last time you will be able to catch me teaching classes live! 

I'm really excited to see you there! Please drop any questions you have about the boot camp in the comments or send me a DM!

taplio

Zach Wilson's Linkedin Analytics

Zach Wilson

Performance ScoreZach Wilson's LinkedIn Performance

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Zach Wilson's Linkedin Analytics

Zach Wilson

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Zach Wilson's LinkedIn Performance

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Zach Wilson's Linkedin Analytics

Performance ScoreZach Wilson's LinkedIn Performance

Zach Wilson's Best Posts (last 30 days)