DataExpert.io

Founder

Illuminate AI

I'm one of the new Bay Area chapter leads for Illuminate AI! 

I help make events better and build this community to the next level!

Chapter Lead

Mage

I'm advising Mage on how to make a data engineering friendly Airflow competitor that will be the next generation of job orchestration!

Advisor

SylphAI

I’ve invested in SylphAI because I’m passionate about building community in AI that helps everyone meet their growth goals and fulfill their potential

Book a call with me at: 

https://app.sylphai.com/user/eczachly/

Airbnb

Changed Airbnb's availability definition from "did a host set a rule that blocked this night" to "can a trip be booked that contains this night" 

Consolidate the availability definition across the company. Saving >500 TBs in the cloud and shrinking the data set to 3% its old size.


Decreased the landing time of Airbnb's unit economics dataset from 3 days to 1 1/2 days. Built a long-term roadmap to further increase the quality of this critical dataset. 

Improved the quality of online systems at Airbnb to be easily compilable by Spark pipelines to compute metrics offline. 

Lead a team of 6 engineers to deliver on the MIDAS initiative to increase the data quality of the Commercial Products org. 

Upleveled smart pricing at Airbnb by improving the feature engineering and latency of the data used to train the smart pricing model.

Staff Data Engineer

Netflix

Built a machine learning feedback system that allowed security engineers to label corporate user behavior as risky or not risky.

Built Asset Inventory - a graph database solution that is a map of all of Netflix's cloud infrastructure.

Senior Software Engineer

I built a pipeline that measures the cloud infrastructure impact on AB tests, saving Netflix millions by allowing them to make smarter AB test rollout decisions.

Senior Data Engineer

Facebook

- Managed a 10 PB+ Hive data warehouse
- Consolidated and conformed company-wide growth metrics (across WhatsApp, Instagram, Messenger, and Facebook) into a single, company-wide view. 
- Optimized machine learning feature set generation pipelines (200+ TB/day) from having a 4 day latency to having a 1 day latency. While also dropping compute costs for those pipelines 4x. 
- Reduced core notification data set latencies from 36 hours to < 8 hours. 
- Migrated 50% of notifications pipelines from using Hive to use Spark, Presto, or real-time streaming.
- Cut compute cost from notifications pipelines by 40% over the course of 9 months.

Data Engineer

Red Pika Labs

I run the technology show for a startup company I helped found called Red Pika Labs. The company tech stack is completely in JavaScript right now. Our first product, Art Lineup (https://www.artlineup.me) launched in March 2017 and has been getting better ever since then!

Tech stack is:
AngularJS frontend on web
React Native frontend on mobile
NodeJS backend
Postgres Database

Research Innovations Incorporated

-Microservices in Node.js and Docker
-RabbitMQ message broker
-Entity extraction and natural language processing with Apache Spark and 3rd party services
-Big data with Accumulo and Hadoop
-Modern front-end systems with Angular.js and data visualizations in D3.js

Software Engineer

Think Big, a Teradata Company

-Implement ETL processes in Hadoop
-build custom MapReduce jobs, Hive UDFs, and Pig UDFs
-design data warehousing approaches with Teradata
-data visualization with Tableau and D3.js
-create data streams using Apache Spark and Apache Storm

I love to teach. I post about #dataengineering and #mentalhealth daily. 

Many years of data engineering experience doing growth analytics at Facebook, security infrastructure at Netflix, and pricing and availability infrastructure at Airbnb. 

I do advanced data engineering boot camps! Join my May 6th cohort!

You can find me by looking up EcZachly on all social networks 

You can search my entire content archive at https://www.dataexpert.io/search

Until it’s our turn, let’s keep clapping for others

I remember feeling embarrassed back when I was in school or as a professional looking for jobs

Everyone else seemed to had it figured out. Extremely successful at whatever they were doing. 

And there I was, feeling like a failure.

But then I realized that everyone lives life at their own pace. What ‘success’ looks like is different for everyone. 

So I started focusing on what I could learn from them, and how I could surround myself with people like them.

I stopped comparing.
I started growing.

And things changed drastically. 

If you’re feeling defeated because of others’ success, change the mindset. Clap for them and learn from their mistakes and wins.

Let’s all be successful, in our own terms.

#StephSynergy

Zach Wilson

I resigned recently from my job at Lockheed Martin :)

It was the best company I had ever worked for and was the first company who showed me what it meant to treat their employees as human first.

I was able to send out 300+ offers and advocate for entry level role candidates.

I was shown how pivotal a FULLY remote and 4 day work week company was and the trust and flexibility that was given to employees. You can truly work anywhere and be productive.

When I gave my notice it wasn’t seen as a betrayal, I wasn’t punished, or locked out of my computer.

Every single teammate reached out to wish me well, say goodbye to me, called me, and thanked me.

They all got together and made me a farewell poster filled with notes.

Even my director reached out and my manager from over a year ago called me.

Thank you Lockheed Martin for allowing me to show up as myself in every single way and nothing less. 

As to what is next, I decided to go all in on myself, within 6 months I scaled my side hustle to over 6 figures and am building a community for introverts and mental health advocacy 🌻 

So thankful for this chapter and so excited for what’s next.

The best is yet to come 🚀

I love Data Engineering!

As a Data Analyst for the past 7 years I find myself loving the work of Data Engineering more and more. I think there is actually a lot of crossover and overlap to Data Analysts as well.

If you can learn how to create simple pipelines from source databases or flat files to a SQL database that adds so much value. Most companies aren't dealing with petabytes of data so keeping things simple and working with things like SSIS, Azure Data Factory, or AWS Glue is a fantastic skill to learn.

Then working on Automations to automatically ingest and append or update that data is the next step.

I think the rise of Data Analysts + Data Engineers mixing to be Analytics Engineers will be a big thing in the future.

AI isn't the cause of the tech hiring slow down! There was a law that went into effect in 2022 that updated Section 174 of the tax code.

Here are two scenarios to illustrate this:

- In 2021, you could found a startup and hire an engineer and pay them $100,000. Say your company did $100,000 in revenue too. In this case, the $100,000 in cost cancels out the $100,000 in revenue, and your business owes zero dollars in taxes. This makes a lot of sense. 

- In 2022, you could found a startup and hire an engineer and pay them $100,000. Say your company did $100,000 in revenue too. Instead of being able to deduct the $100k salary in the first year, you have to amortize the $100k salary over 5 years. 
So in this case, you'd have a taxable income of $80k for the first year. This means you'll have a $24k tax bill with no money to pay it! 

Many companies were caught off guard in 2022 and 2023 during tax season and had to take out loans to pay their tax bills!

So nowadays, when hiring a software engineer companies have to think, "will this hire pay off in 5 years?" and the answer when hiring junior engineers is, "maybe, maybe not." 

In the old world, it didn't matter because 100% of the junior engineers salary could be deducted from a businesses taxes! 

If you're struggling to get a job in tech, please call your senator and demand that they repeal the update to section 174 of the tax code! This would immediately solve about 50% of the tech hiring slow down we've seen in the last 2 years!

#softwareengineering 

The May 6th boot camp will be my last one because I landed a job at OpenAI! 

I will be growing their data engineering team. 

Given this new responsibility, I’ll be toning down the content. 

- No more spicy memes (sorry for letting you down Carly!) 
- No more hot takes
- No more posts at all 

I’ll be moving to the forest and deleting LinkedIn so I can focus on building AGI to deprecate data engineering!

Thank you all so much for the valuable training data you’ve supplied me over the last 3 years as a content creator!

#dataengineering

I love Data Analytics! 

As a Data Engineer for the past 9 years I find myself loving the work of Data Analytics more and more. I think there is actually a lot of crossover and overlap to Data Engineer as well.

If you can learn how to create clear dashboards from SQL queries it adds so much value. Most companies aren't dealing with petabytes of data so keeping things simple and working with things like Tableau, Excel, or Power BI is a fantastic skill to learn.

Then working on metrics to guide business decisions and measure impact with experiments is the next step.

I think the rise of Data Analysts + Data Engineers mixing to be Analytics Engineers will be a big thing in the future.

Inspired by Alex Freberg

I DID IT!!

I’m happy to share that after completing a bootcamp as a professional student (if that’s a thing), I will now switch hats to be their new LinkedIn Instructor!

When Zach Wilson brought this opportunity to my attention, I couldn’t believe it. 

I’m excited to bring all my knowledge and experiences building my brand on this platform to the new group that will be embarking in this journey next month.

This happened in a very unique way. I enrolled as a student while being a professional - and this was definitely a challenge. I struggled to understand how to even start. The journey was full of challenges. 

I remember setting meetings with the instructor to get my platforms (SQL, GitHub, etc.) ready to go and things just seemed to not work the way they were supposed to. But I didn’t give up. I wanted to understand how the world of data and AI worked together. I wanted to be 1% better than what I was yesterday. 

I showed up to every class, asked questions (way too many), and connected with others that were more advanced so I could get myself to the level I wanted to be at.

And I did it. I completed that and felt a sense of fulfillment for not giving up. For me, it was more about the journey and the lessons learned. It was the knowledge acquired. I can proudly say this experience changed a portion of my career. 

Now, I will get to sit on the same classroom, but teaching others the power of LinkedIn. A full circle moment.

Sharing resources and helping others is my passion. If you’re a professional interested in developing your skills in Data, LLMs or AI, and also want to learn all things LinkedIn (with me), here’s a link to the same course/bootcamp I completed: https://lnkd.in/guZdAgu5.

I wanted to share my story, and also encourage you all to go after those “difficult” goals. No matter how hard they seem to be, keep going. 

You will thank yourself for not giving up once you get the ‘prize’ and reach that final destination. 

#StephSynergy

There's no better feeling than closing all your browser tabs after finally solving a bug

Join my free Software Engineering newsletter: https://www.thesweekly.com

#coding #programming #softwareengineering 

Here's what the average data engineering interview looks like in 2024:

- 1 hour algorithms in Python
Here you will be asked irrelevant questions about dynamic programming, linked lists, and inverting trees

- 1 hour SQL 
Here you will be asked niche questions about recursive CTEs that you've used once in your ten year career

- 1 hour data architecture
Here you will be asked about CAP theorem, lambda vs kappa, and a bunch of other things that ChatGPT probably could answer in a heartbeat

- 1 hour behavioral
Here you will be asked about how to play nicely with your coworkers. This is the most relevant interview in my opinion

- 1 hour project deep dive
Here you will be asked to make up a story about something you did or did not do in the past that was a technical marvel

- 4 hour take home assignment
Here you will be asked to build their entire data engineering stack from scratch over a weekend because why hire data engineers when you can submit them to tests? 

#dataengineering 

You don't want to wake up in 10 years wishing you would have taken action today. 

That's how I've been living my life for the past 7 years. Before that I just with the flow and didn't really care what happened. Now every day I try to set goals that I'll be proud of in 10 years.

I can look back and say I don't have any regrets. That I took the risk, I worked hard, and I made my family proud.

I don't know who needs to hear that, but that mentality changed my life.

Here Comes the Gold mine for Data Engineers

Python Complete Playlist (5 videos already released)
lecture 1 - https://lnkd.in/gh_gVsTm
lecture 2 - https://lnkd.in/gCxba2zB
lecture 3 - https://lnkd.in/g6DVh4va
lecture 4 - https://lnkd.in/gs9TFEWK
lecture 5 - https://lnkd.in/gFPaaAM6

Free SQL Complete Course (one shot video 9 hours)
https://lnkd.in/gXTvqk3i

Data Engineering Mock Interviews (Data Engineers Club)
Interview 1 - https://lnkd.in/gjascwBm
Interview 2 - https://lnkd.in/g7AB8TNt
Interview 3 - https://lnkd.in/gzZBRcKu
Interview 4 - https://lnkd.in/gx9kg95h
Interview 5 - https://lnkd.in/gQ9kVUrf
Interview 6 - https://lnkd.in/gbxkjudi
Interview 7 - https://lnkd.in/g9e_Kiks
Interview 8 - https://lnkd.in/gshc46Pq
Interview 9 - https://lnkd.in/gfpNwNZK
Interview 10 - https://lnkd.in/gpchPS32

Everyday we are releasing one mock interview at 7 pm on my youtube channel.

Also more videos will be coming in the python playlist.

Do subscribe to my channel so that you do not miss out on anything.

Link to the channel - https://lnkd.in/gun_wZeK

Also, just to let you know that my new big data batch is starting tomorrow. To check more about my big data program you can visit https://lnkd.in/gZp_nBNR

Happy Learning!

#bigdata #dataengineering #interview #apachespark #sql #python

Hiring managers, I need to have a word with y’all. 

Those of us who are lucky enough to make hiring decisions literally have people’s livelihoods in our hands.

Every time you decline a resume, there’s a real person on the other end receiving that rejection.

Imagine seeing an email from a place you applied and how that experience can go from excitement to gut punch really quickly. 

And sadly, yours will probably not be their first rejection that week. 

So we should all do well to remember that there are real people with hopes, dreams, and worries on the other end of the application.

I don’t want to data model anymore. I want to date a model!

#dataengineering

There’s a lot of noise in data engineering! 

Remember if something seems “shiny” there’s a good chance it’s not “safe”

Learn data modeling and analytical patterns, not some LLM-generated garbage that false promises to eliminate data modeling 

Learn distributed compute with Spark or BigQuery or Snowflake, not some niche technology that isn’t used much! 

Learn Python, not Rust or Mojo

The key is to build a resume of languages and technologies that are actually used in industry! 

If you’re already in industry, maybe learn a higher paying niche. If you’re breaking in, it’s a waste of time! 

#dataengineering

Data analytics is going to become more "Kafka-first" for a variety of reasons

- Relying on a data engineer to ETL the data is a bottleneck that a lot of companies don't want to worry about

- Technologies like Apache Pinot sit on top of Kafka and enable real-time analytics without ETL and without data engineers

- Replicating Kakfa data to the data lake for long term storage is as easy as defining a schema and clicking a few buttons in Confluent.

I've been very impressed with the longevity of Kafka. I see it becoming a BIGGER player in the coming years instead of ramping down like you'd expect from a thirteen year old technology!

#dataengineering

I’m really glad I got to meet up with Aishwarya Srinivasan!

She is one of those prolific people who creates so much! 

- she wrote a book on personal branding recently!
- she leads Illuminate AI, a company for AI mentorship  
- she works for Microsoft looking at GenAI startups 
- she creates content on many platforms like me 

It’s really great when you get to hang out with inspiring people like Aishwarya! 

#artificialintelligence

There are 4 levels to being data scientist!

Inspired by Brad’s post a week ago 


- level 1
You know SQL and Python and how to clean data 

- level 2 
You know experimentation, Ab testing and everything product analytics 

- level 3 
You’re a machine learning beast 

- level 4 
You know when to not use machine learning!


#datascience

4 Reasons I won't be going back to get a Masters Degree even though I have a Bachelors in Recreational Therapy:

1. No One Cares about my Degree Now
In my consulting work I consult with tech companies and startups on all sorts of projects. I haven't ever had 1 ask about my degree. They care about my experience.

2. I don't want to Climb the Corporate Ladder
If I wanted to be a CTO or CEO of a larger company I may have to go back and get an MBA or a Masters in Finance or Computer Science. I was an Analytics Manager and that experience made me not want to climb higher on the corporate ladder.

3. Brand Recognition
I get 90% of my consulting work from people recognizing me from YouTube, LinkedIn, and other places. They see my tutorials on things they're having trouble with (SQL, Data Migration, Data Cleaning, Azure, etc.) and reach out to see if I can solve their problem.

4. Time and Cost
Going back to get a Masters takes time (1-2 years minimum) and money (50k+ per year). I could do this while I'm doing all my normal work, but it would be quite extreme to be able to fit that into my current workload. Most likely I would have to take on less consulting work and work less on my content for YouTube and Analyst Builder. That doesn't sound like a good tradeoff.

This was the best creator meetup I’ve been to in a long time! 

Here’s the squad: 
Alex Xu  - makes the best system design content! Teaches millions through his company ByteByteGo  

Chip Huyen - her book on ML systems is 🔥. Also check out her company Voltron Data! 

Marily Nika, Ph.D - her Maven course on AI product management is incredible!

Aishwarya Srinivasan - her new book on personal branding is amazing! 

Daliana Liu - her podcast The Data Scientist show is so insightful! I’ve been on multiple times! 

Ryan Peterman - has a valuable newsletter on growing your software career, developing.dev 

Carter Abdallah - makes great content on Instagram and TikTok about software engineering

Zach Wilson  - makes delightful data engineering content and teaches through his company DataExpert.io 


#dataengineering 
#softwareengineering 
#datascience
#machinelearning

A data pipeline is more than a regularly refreshing SQL query!  

Pipelines should be resilient along many dimensions: 

- pipelines shouldn’t break when data distribution changes 
Gracefully managing skew is an important part of this journey. 

- pipelines shouldn’t break when ownership changes 
Documented pipelines have much longer lifetimes than undocumented ones!

- pipelines should be able to be updated with new fields and increased requirements 
Adding the proper staging steps and upstream combinations will allow for easier change management of your pipeline. 

- pipelines shouldn’t publish bad data!
Bad data should be caught before it’s published. Write-audit-publish pattern is the best way to solve this! 

How else should your data pipelines be resilient?

#dataengineering

Taplio

Zach Wilson's Linkedin Analytics

Zach Wilson

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Zach Wilson's Linkedin Analytics

Zach Wilson

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts .css-qqfgvy{font-size:var(--chakra-fontSizes-md);}(last 30 days)

Want to drive more opportunities from LinkedIn?

Zach Wilson's Linkedin Analytics

Zach Wilson's Best Posts (last 30 days)