DataExpert.io

Founder

Ardent AI

I think AI will disrupt data engineering in a significant way. I’m investing in Ardent to accelerate that disruption!

Angel Investor

Peony Health

I invested in Peony Health because I believe habits are greatly impacted by neurochemistry and Susie Chen is a technical founder with business chops who will deliver on their mission

Illuminate AI

I'm one of the new Bay Area chapter leads for Illuminate AI! 

I help make events better and build this community to the next level!

Chapter Lead

Mage

I'm advising Mage on how to make a data engineering friendly Airflow competitor that will be the next generation of job orchestration!

Advisor

A House in Utah

Former Child

SylphAI

I’ve invested in SylphAI because I’m passionate about building community in AI that helps everyone meet their growth goals and fulfill their potential

Airbnb

Changed Airbnb's availability definition from "did a host set a rule that blocked this night" to "can a trip be booked that contains this night" 

Consolidate the availability definition across the company. Saving >500 TBs in the cloud and shrinking the data set to 3% its old size.


Decreased the landing time of Airbnb's unit economics dataset from 3 days to 1 1/2 days. Built a long-term roadmap to further increase the quality of this critical dataset. 

Improved the quality of online systems at Airbnb to be easily compilable by Spark pipelines to compute metrics offline. 

Lead a team of 6 engineers to deliver on the MIDAS initiative to increase the data quality of the Commercial Products org. 

Upleveled smart pricing at Airbnb by improving the feature engineering and latency of the data used to train the smart pricing model.

Staff Data Engineer - TC $600k ($220k Salary $320k Stock $60k bonus)

Netflix

Built a machine learning feedback system that allowed security engineers to label corporate user behavior as risky or not risky.

Built Asset Inventory - a graph database solution that is a map of all of Netflix's cloud infrastructure.

Senior Software Engineer - TC ~$575k ($550k salary + 5% options)

I built a pipeline that measures the cloud infrastructure impact on AB tests, saving Netflix millions by allowing them to make smarter AB test rollout decisions.

Senior Data Engineer - TC ~$365k ($365k salary + 5% options)

Facebook

- Managed a 10 PB+ Hive data warehouse
- Consolidated and conformed company-wide growth metrics (across WhatsApp, Instagram, Messenger, and Facebook) into a single, company-wide view. 
- Optimized machine learning feature set generation pipelines (200+ TB/day) from having a 4 day latency to having a 1 day latency. While also dropping compute costs for those pipelines 4x. 
- Reduced core notification data set latencies from 36 hours to < 8 hours. 
- Migrated 50% of notifications pipelines from using Hive to use Spark, Presto, or real-time streaming.
- Cut compute cost from notifications pipelines by 40% over the course of 9 months.

Data Engineer - TC $185k ($135k salary, $50k stock)

I love to teach. I post about #dataengineering and #mentalhealth daily. 

Many years of data engineering experience doing growth analytics at Facebook, security infrastructure at Netflix, and pricing and availability infrastructure at Airbnb. 

You can find me by looking up EcZachly on all social networks 

Join my Data Learning Platform https://www.dataexpert.io

Data modeling needs an update in 2025! 

Slowly-changing dimensions add tons of engineering complexity by saving a tiny amount storage and compute. 

We live in an era of extremely cheap storage (S3 glacier flexible storage is $44 per TB per year!) 

The two weeks of data engineering time you used to build SCD type 2 cost you $10,000. 

This means you’ll break even in 227 years if SCD type 2 reduces 1 TB of space. (still better than TSLA’s PE ratio)

Meta doesn’t even use SCD type 2 anymore. They just do daily snapshots of their dimensions. It’s way less brittle and more flexible to change! 

Here’s the 2000 word detailed article:  https://lnkd.in/g_5SxveH to read more about Meta’s approach and what a new era data lake should look like! 

Thanks to Sahar Massachi for collaborating with me on this article. He’s open to new full time roles in New York City right now. 

He and I did some of really impactful data engineering work at Meta together!

Zach Wilson

Data Engineering ​> AI Engineering and I’m tired of pretending it isn’t

I love Data Analytics! 

As a Data Engineer for the past 9 years I find myself loving the work of Data Analytics more and more. I think there is actually a lot of crossover and overlap to Data Engineer as well.

If you can learn how to create clear dashboards from SQL queries it adds so much value. Most companies aren't dealing with petabytes of data so keeping things simple and working with things like Tableau, Excel, or Power BI is a fantastic skill to learn.

Then working on metrics to guide business decisions and measure impact with experiments is the next step.

I think the rise of Data Analysts + Data Engineers mixing to be Analytics Engineers will be a big thing in the future (and the present)

AI has changed how you break into data engineering a little bit but the fundamentals that are tested in interviews still matter a lot!

You still have to know:

- SQL
- Python/Scala
- Distributed Compute 
- Data Modeling
- Building an awesome portfolio project
- Job Networking 

In this 3500 word article, I cover everything in excruciating detail:

https://lnkd.in/gZssYS5Z

This chart seems extremely alarming. Here are 3 reasons AI isn’t “taking all the jobs” 

1. Job openings are still a bit higher than 2019 back before zero interest rates were introduced. The job market isn’t “bad” right now. It’s returning back to pre-COVID levels which is normal  

2. ChatGPT was really bad when it was released and wasn’t “automating” anything with GPT 3.5 turbo. Yet job openings were still being crushed anyway 

3. Interest rates soared in 2022 which forced companies to tighten their belts. The chart is waaaay more correlated with interest rates than AI capabilities

DataExpert.io made $168,510 in October 2025

- $93,553 from AI cohort sales 
- $61,000 from DataExpert.io subscriptions 
- $6,000 from Blog.dataexpert.io subscriptions 
- $5,000 from newsletter sponsorships (new)
- $1733 from TikTok
- $987 from YouTube 
- $237 from X

I had $24,350 in costs: 

- $20,347 in headcount 
- $4,003 in cloud costs (AWS, Databricks, OpenAI, Snowflake and Astronomer) 

Currently operating at: 85.7% profit margins which is up from 57% in September mostly driven by downsizing headcount ($35.3k-> $20.3k) and increased sales ($82k -> $168k)

Founder @ DataExpert.io | ADHD

Imagine applying for 400 jobs, landing 1 interview and then being asked an irrelevant leetcode question that AI could solve in 5 seconds and getting passed over! 

That’s the 2025 job market right now. Hang in there!

#softwareengineering 
#dataengineering

I let early retirement dreams destroy my 20s! 

When everybody else was taking dream vacations and traveling the world, I didn't leave the USA until I was 28 years old. 

When everybody else was living in nice apartments, I set a strict budget of $1500/month for rent in San Francisco (the quality sacrifice here is real). 

When everybody else was finding their life partners, I was overly focused on earning more by having my eyes bleed in front of a screen

These "sacrifices" used to be things I looked at with pride. "Look how long term-minded I am being" 

Now I look at it very differently now. 

Early retirement is a scam. There won't be a day when you're 35 when you're like... that's enough work for my entire life. 

CoastFIRE is what I needed to shoot for, not FatFIRE at 30. 

CoastFIRE is where you stop investing in the future because you have a big enough nest egg to retire. You just have to make enough money to cover your expenses so you don't touch it. I spend ~$80,000/year in San Francisco. So I can take a MUCH LESS stressful job and just vibe. 

CoastFIRE goals are much easier to obtain. If you can have $300k saved for retirement by 30, you can not investment another dollar and have a comfortable retirement. These goals are much more reasonable than "I MUST BE A MILLIONAIRE BY THIRTY OTHERWISE I WILL LITERALLY DIE" 

And I threw away the best years of life staring in front of a screen. Years I could have spent falling in love, traveling the world, and enjoying. Instead I get to be stuck here by myself at 31 with a mountain of money and nobody to share it with. 

I would've done things differently if twenty year old Zach had the same knowledge that thirty year old Zach has now

Fifteen years ago, I was praying I didn’t fail my Microsoft Excel exam

In 2025, I use SQL so much I would fail that same exam

A mistake many data engineers make is thinking real-time = streaming pipelines! 

Whenever a stakeholder says they want data in real time, you shouldn't default to dreams of Flink, Kafka, and watermarks. 

You should clarify with precision what an acceptable amount of latency is for this use case. 

Many times when a stakeholder asks for real-time data, it can be solved with an hourly batch pipeline. The incremental benefit to jump from hourly batch to streaming isn't worth it because it impacts the homogeneity of your suite of pipelines and makes the overhead maintenance much higher! 

Sometimes stakeholders say real-time and what they mean is "predictable refresh rates." This is a sign you need to do better as a data engineer at setting SLAs for your pipelines about when they'll refresh.

SparkSQL is often superior to Spark Dataframes or Spark Datasets!

Why is that the case? 

- SparkSQL is ANSI compliant. If you ever want to migrate off of Spark, copying and pasting the query to the new engine will work a lot of the time! (minus Spark-specific functions)

- SparkSQL is highly optimized and will outperform PySpark data frames (and match Scala data frames) in performance matchups.  

- SparkSQL is a better common denominator than Dataframes or Datasets because data scientists and other less engineering-focused individuals can also update and tweak queries!

Master the interplay between pain, passion, pleasure and play and change your life forever!

Here’s why cohort education isn’t for me and what I’ll be doing next after April 2026: 

- I want to raise prices but also being perceived as a scammer is something that bothers me a lot when I genuinely want what’s best for students and genuinely design my program to have a high ROI 

- I want to swap in other instructors but that has come at a big loss in customer satisfaction

So it seems like what customers want is:

Low prices, high quality and Zach running into the ground by doing everything himself. 

So I’ve been doing exactly that for 2 1/2 years and it’s made me millions of dollars but customers are unwilling to budge. 

This cohort hamster wheel isn’t something that is rewarding me like it used to. Could I do this for the rest of my life and retire with 100 million dollars? Sure. Is that a life worth living? Not for me. 

A lot of you are asking what I’ll be doing next. 

And here’s the honest truth: 

Resetting my nervous system. Ever since I was homeless and addicted in 2011, I made a promise to myself to never be in that position ever again, I’ve been running from despair. And now I have millions of dollars and I don’t need to run anymore. 

Right now when I’m not running, I feel like I’m dying and becoming a loser again. I cannot sit still. Whenever I’m resting, I feel worthless. 

I don’t need to run AT ALL. I need to ENJOY and not let my childhood trauma dictate the trajectory of my life.  

It’s been a GLORIOUS 14 year run since 2011. I’ve worked at every big tech company and became internet famous but my inner child is screaming, “let me rest!” 

I’ve made my teenage-self proud and it’s time to stop running. It’s time to reset my nervous system and let it be okay with resting and enjoying. 

When I have a healthy relationship with rest (something I’ve had for zero percent of my life), that is when I’ll let you guys know “what I’m doing next.”

After DataExpert.io boot camp, Hitesh Kaushik almost immediately doubled his salary. Now he's leaving India to Dubai and tripling it again! 

Six times more money in less than a year! That's what can happen if you dedicate yourself to learning!

To celebrate Hitesh and Diwali, you can get 50% off my newsletter FOR LIFE here: https://lnkd.in/e2FYu9AS

Only 50 days left in 2025. 

Most people will coast. A few will finish strong.

Mitali Gupta and I are launching the 50-Day Data Challenge to help you end the year sharper than you started.

Every day from now until December 31, you’ll get one of the following:
- A 5–10 question quiz to test your knowledge
- A diagram upload graded by AI to train your architecture and modeling
- A SQL coding challenge to ramp up your analytical skills

Each challenge earns you a DataExpert.io badge — and if you complete all 50, you’ll unlock discounted access to our cloud subscription product. 
The first two people to finish the challenge will get one year access for free! 

We start today!

Are you in? 

Comment “I’m in”, repost and tag a friend to join you.

Sitting for >4 hours per day is as bad for you as smoking a pack of cigarettes! 

Make sure to fight back the “desk worker head tilt” 

Break your days up
Use a standing desk 
Stretch your back and neck regularly
Take walking meetings 

Stop slowly dying at your desk! 

What else do you do to make your life movement-oriented?

After I graduated in 2014, I had two job offers in front of me. 

1. Work at Hill Airforce Base. Coding in C and Lua on weapons systems as a software engineer. Base salary: $60k
2. Work at Savvysherpa, Inc. as a data science intern. Base salary: $40k 

I picked the second option because I believed in data science as a trend and the learning opportunities offered to me.

Do I miss that $20k I could’ve made? Of course not! I caught one of the hottest technology trends pretty early on and my salary grew way faster than had I taken the government job

Always optimize your early career for learning over earning!

SQL is deceptively complex. The order which things apply isn’t that intuitive and can be frustrating when debugging queries. 

Let’s talk about the ordering of a query and when each step is executed. 
Here’s the query well deconstruct. 

SELECT 
 city, 
 SUM(weed_smoked) as total_weed_smoked 
FROM weed_consumption 
WHERE is_legal = TRUE 
GROUP BY city 
HAVING SUM(weed_smoked) > 420
ORDER BY total_weed_smoked DESC 
LIMIT 69

First, SQL executes FROM and WHERE. So we take the weed_consumption table and filter to only rows where is_legal is True. 

Then we GROUP BY city to start our aggregation. 

Then we filter the aggregation to only cities that have smoked at least 420 pound of weed. HAVING only ever applies to the aggregated result. 

Then we apply SELECT and aliases.

After that we ORDER BY so we see the most Snoop-like cities first. 

Then we LIMIT to the top 69 cities.

Learning patterns and tricks can really elevate your  data engineering! 

- the sorted-merge-bucket join (SMB join) 
This type of join requires both the left and right tables to be sorted and bucketed on the join key. If they are, this join can happen without shuffling and is extremely fast. I used this technique at Facebook to save tens of thousands of CPU days. 

- the datelist data structure 
Sometimes having a partial history in the same row can dramatically increase performance. At Facebook, I used this concept to store the last 30 days of someone’s activity as an integer. 01010111… the first place is their activity today. The last place is their activity 30 days ago. This reduced representation can have a huge impact on performance.  

- write-audit-publish pattern 
This one is critical for data quality. This pattern treats publishing to production as a contract. Write to a staging table, run your quality checks, if they pass, move the data from staging to production. 

- idempotent ETLs 
Writing ETLs that generate the same data regardless of if you run them today or next week is very useful. Avoid current time stamps, unbounded date ranges, and non-parameterized filtering. Always have a “logical” date that filters the data sets you’re processing. Following this pattern makes backfilling much easier.

AI engineers spend 80% of their time getting the context right is the same as data scientists spending 80% of their time cleaning data

No matter how fancy the algorithm is, if it’s fed bad information, you’re going to get a bad result!

I applied for Y Combinator's 2026 winter batch. Exciting to think what might happen if I get accepted!

taplio

Zach Wilson's Linkedin Analytics

Zach Wilson

Performance ScoreZach Wilson's LinkedIn Performance

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Zach Wilson's Linkedin Analytics

Zach Wilson

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Zach Wilson's LinkedIn Performance

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Zach Wilson's Linkedin Analytics

Performance ScoreZach Wilson's LinkedIn Performance

Zach Wilson's Best Posts (last 30 days)