EcZachly Inc

Founder

SylphAI

I’ve invested in SylphAI because I’m passionate about remote work and machine learning.  

Book a call with me at: 

https://app.sylphai.com/user/eczachly/

SylphAI is an AI-driven project staffing and collaboration platform, empowering a global AI community.

Advisor

Mage

I'm advising Mage on how to make a data engineering friendly Airflow competitor that will be the next generation of job orchestration!

Airbnb

Changed Airbnb's availability definition from "did a host set a rule that blocked this night" to "can a trip be booked that contains this night" 

Consolidate the availability definition across the company. Saving >500 TBs in the cloud and shrinking the data set to 3% its old size.


Decreased the landing time of Airbnb's unit economics dataset from 3 days to 1 1/2 days. Built a long-term roadmap to further increase the quality of this critical dataset. 

Improved the quality of online systems at Airbnb to be easily compilable by Spark pipelines to compute metrics offline. 

Lead a team of 6 engineers to deliver on the MIDAS initiative to increase the data quality of the Commercial Products org. 

Upleveled smart pricing at Airbnb by improving the feature engineering and latency of the data used to train the smart pricing model.

Staff Data Engineer

Netflix

Built a machine learning feedback system that allowed security engineers to label corporate user behavior as risky or not risky.

Built Asset Inventory - a graph database solution that is a map of all of Netflix's cloud infrastructure.

Senior Software Engineer

I built a pipeline that measures the cloud infrastructure impact on AB tests, saving Netflix millions by allowing them to make smarter AB test rollout decisions.

Senior Data Engineer

Facebook

- Managed a 10 PB+ Hive data warehouse
- Consolidated and conformed company-wide growth metrics (across WhatsApp, Instagram, Messenger, and Facebook) into a single, company-wide view. 
- Optimized machine learning feature set generation pipelines (200+ TB/day) from having a 4 day latency to having a 1 day latency. While also dropping compute costs for those pipelines 4x. 
- Reduced core notification data set latencies from 36 hours to < 8 hours. 
- Migrated 50% of notifications pipelines from using Hive to use Spark, Presto, or real-time streaming.
- Cut compute cost from notifications pipelines by 40% over the course of 9 months.

Data Engineer

Red Pika Labs

I run the technology show for a startup company I helped found called Red Pika Labs. The company tech stack is completely in JavaScript right now. Our first product, Art Lineup (https://www.artlineup.me) launched in March 2017 and has been getting better ever since then!

Tech stack is:
AngularJS frontend on web
React Native frontend on mobile
NodeJS backend
Postgres Database

Research Innovations Incorporated

-Microservices in Node.js and Docker
-RabbitMQ message broker
-Entity extraction and natural language processing with Apache Spark and 3rd party services
-Big data with Accumulo and Hadoop
-Modern front-end systems with Angular.js and data visualizations in D3.js

Software Engineer

Think Big, a Teradata Company

-Implement ETL processes in Hadoop
-build custom MapReduce jobs, Hive UDFs, and Pig UDFs
-design data warehousing approaches with Teradata
-data visualization with Tableau and D3.js
-create data streams using Apache Spark and Apache Storm

Savvysherpa, Inc.

- Create data visualizations using Tableau 
- Database management and migrations
- Data mining in Python
- Data analysis and management using SQL 
- Setting up and managing graphical databases such as OrientDB and AllegroGraph
- Automate ETL processes using Pentaho and SQL

Data Science Engineer

Data visualization using Tableau and data analysis using SQL and R.

Data Science Intern

I love to teach. I post about #dataengineering and #mentalhealth daily. 

Many years of data engineering experience doing growth analytics at Facebook, security infrastructure at Netflix, and pricing and availability infrastructure at Airbnb. 

Subscribe to my YouTube channel, Data with Zach! 
https://www.youtube.com/c/DatawithZach
https://www.tiktok.com/@eczachly

You can find me by looking up EcZachly on all social networks 

You can search my entire content archive at https://www.zachwilson.tech/search

Fundamental concepts every data engineer should know because they don’t really change

- ANSI SQL 
- distributed compute 
- OLTP vs OLAP 
- CAP theorem 
- slowly-changing dimensional modeling 
- fact data modeling
- logging best practices 
- AVRO / Thrift schemas
- idempotent pipelines 
- job orchestration 
- flexible schema vs defined schema 
- data quality testing 

#dataengineering

Zach Wilson

Data Engineering Demystified | ADHD | Advisor@Mage,SylphAI | @EcZachly on TT,IG,Twitter | 250k+ followers

Curious where people were before data engineering. I started as a BI analyst at Savvysherpa, Inc.  

#dataengineering

Whenever you’re learning something new, your brain often gives you anxiety because of the unfamiliarity of the topic. 

The reason for this is unfamiliarity back in the day was a death sentence. 

An unfamiliar snake in your jungle hut = a near death experience. 

Learning an unfamiliar snake language like Python = the brain can’t tell the difference and freaks out. 

Give yourself grace and a little bit of courage when approaching a new topic or idea!


#datascience

Big data SQL JOINs with high cardinality have a few tiers of size: 

<10 million rows = query comes back almost instantly 
<100 million rows = query comes back in a few minutes 
<1 billion rows = query comes back in an hour or two 
<10 billion rows = query takes half the day 
<100 billion rows = query takes a few days 
<1 trillion rows = query laughs at you and your infinite cloud budget 

Broadcast join is your friend until it doesn’t fit into memory, then you become Jeff Bezos best friend! 

#dataengineering

When I worked at Facebook, data engineers weren’t considered engineers. They were considered individual contributors! 

Facebook at the time had two tracks. The engineer track for software engineers and the individual contributor track for everybody else. 

The only real difference between these tracks was compensation. Engineers got ~30-50% more stocks as individual contributors at the same level. 

This was something that made me so mad because I had all the technical skills that the software engineers had and was doing very technical data engineering work as well!

Netflix and Airbnb treat their data engineers differently. They get paid equally to the software engineers at the same level. 

This illustrates how Facebook data engineers are closer to data analysts and analytics engineering whereas Netflix and Airbnb data engineers are considered to be like software engineers! 

#dataengineering 
#softwareengineering

LLMs are going to make data modeling an overpowered data engineering skill! 

If you can describe the input schemas and the output schemas of your pipeline, LLMs can generate the SQL details and Airflow DAG steps that are in between! 

I noticed when working with ChatGPT that it does much better at generating pipelines when you supply the expected output schema as a "nudge" in the right direction. 

If you just supply the input schemas and analytical pattern (slowly-changing dimension, cumulation, etc), it doesn't quite know what to output and the results are kind of garbage. 

Definitely try this when writing your next pipeline! You'll be surprised how much faster you are! 

#dataengineering 


Google got a nice branded word for search. What should ChatGPTs be?
#machinelearning

Things I’ve learned running my data engineering boot camp for 2 weeks! 

- don’t use platforms like Maven which take 10%. Google classroom is free and good enough. 

- be more than 1 day ahead of the boot camp. When curricula building you should be working on the following weeks curricula this week so your students can have access to the materials before hand. 

- use Postgres 14+ when running your SQL sections otherwise you don’t have access to BIT_COUNT function 

- use 1 long-standing Zoom call instead of 1 Zoom call per session. This makes it easier for students to know where to go and don’t use the Google Calendar + Zoom integration, it’s really bad! 

- give people more time on the homework. Strict deadlines are for college not boot camps. Most of the people in my boot camp have full time jobs 

- record each session with Loom. Loom’s recording for 2 hours is ~2 GBs and MUCH higher quality than the recording Zoom gives you. 

- paste intermediate SQL queries in the Zoom chat so attendees don’t feel so much pressure to keep up coding with me 

- operate about 30-40% slower to communicate the information more clearly and effectively


#dataengineering

Data architects usually make more money than data engineers because: 

Building scalable architectures enable analysts, scientists and other engineers to more quickly answer questions across a large swathe of data, not just the outputs of a single pipeline. 

Enforcing best practices is a scalable way to impact the productivity of the rest of the engineers in your team. Leaders are a force multiplier. 

Bad data governance is a massive legal risk. Great data architects work with data infrastructure teams to build tooling that buys down this risk. 

#dataengineering 
#dataarchitect

EcZachly Inc is starting to feel a bit more official! #dataengineering

Here are some boundaries that changed my life and my mental health! 

- no phone after 10 PM. I used to scroll mindlessly until 1 AM which impacted my sleep. 

- video games with Wesley but only for 60-90 minutes. I used to game for 5-6 hours straight and waste too much time. 

- no running or exercise after 8 PM. I used to run from 10PM to 11:30 PM and it impacted my sleep. 

- no phone for the first hour of the day. This gives me a deeper sense of control! 

#mentalhealth

The future workflow of data engineers will look like this: 

- Source data extraction gets automated with things like Mage. There's an extremely large but finite number of connectors that you need to implement and Mage pretty much has all of them. 

- Logging specifications happen in collaboration with software engineers. These input schemas feed a large number of downstream pipelines 

- 3rd-party API sources, logs, and production database snapshots account for almost all the data you need in your warehouse. Once these sources are in the lake, that's when LLMs come into play. 

- In the future, special-purpose LLMs will be able to look at all your sources and your expected output data models and give you the most "robust" data model that will answer all the questions you need answered while being as efficient as possible. Companies like Tobiko Data are working on this problem too from a SQL perspective. 

- That robust data modeling will greatly enhance our understanding of the business environment and allow businesses to experiment and tinker with things at a much faster rate. 

So what does a data engineer need to know given these turns of events?

- Conceptual knowledge is more important than extremely detailed knowledge. Learning a bunch of things more shallowly will serve you more than learning a few things deeply in the future. 

- A focus on data modeling, business impact, and communication will be the skills that are hard for LLMs to replace because they're more matters of empathy and understanding. Focusing deeply on these will help you make LLMs your friend, not your doom!

What else should data engineers be aware of given all the recent changes in the technological landscape? 

#dataengineering 

Thousands of people have applied to my boot camp so far. One of the questions I ask is "What topic excites you the most?"  

Seems like data engineers are starting to see that data modeling matters a lot! Although Spark and Flink are very close too! 

If you want to apply to EcZachly Inc's next boot camp iteration, please fill out this form: https://lnkd.in/gAxWPDis

#dataengineering 

3 data podcasts that are really good: 

- Ask A Data Mentor Podcast 
Hosted by Sarah Floris, MS. She recently started and the content is already really good! 

- Ken's Nearest Neighbors 
Hosted by Ken Jee has tons of episodes. I’ve been on twice! 

- The Data Scientist Show 
Hosted by Daliana Liu. My episode on data engineering is one of the most viewed episodes! 


#dataengineering 
#datascience

It took me like 5 minutes to understand this stats joke but once I did, I died 

#datascience

Some unstoppable trends: 

- tech layoffs creating a surge in flexible work and solopreneurship with platforms like SylphAI and topmate.io 

- data engineering becoming both more software engineering-focused as data products become more mainstream 

- simpler data engineering becoming less software focused with low-code solutions like Fivetran 

- Airflow becoming the “weakest link” in the data engineering stack and replaced by popular competitors like Mage, Prefect and Dagster 

- MLOps getting a lot more attention as more and more companies learn the pains of running a model in production 

- MLOps, DataOps, data pipeline and ML training tooling see a unification when people realize they are strongly connected 

Any MLOps companies you know that deserve a shoutout?

 #dataengineering 
#machinelearning

Reminder that Lulu’s 10th birthday party is tomorrow from 6-8 PM in Mission Dolores Park! 

I hope to see you there if you can make it! Bring your dog and let’s talk data, careers, and networking!

#dataengineering 
#datascience

Open article: Dogs and Data, Lulu’s 10th birthday!! by eventbrite.com

https://www.eventbrite.com/e/dogs-and-data-lulus-10th-birthday-tickets-617250140787

Lulu’s party yesterday was amazing! 

Thank you everybody who showed up even though it was a little rainy and windy! 


Reshma, Vidya, Meri

When writing window functions in SQL, I always use DENSE_RANK and ROW_NUMBER. 

I never use RANK. 

What are the use cases for RANK? Is it like JavaScript’s double equal sign that looks right but almost never is?

#dataengineering 
#sql
#softwareengineering

One year ago today, I published my compensation history in an effort to combat taboos. 

I’m grateful to see the legal progress made in states like Colorado and California. 

Although some of the salary ranges I’ve seen are straight up ridiculous. Looking at you Netflix with your $150k to $900k postings for data engineering! 

#dataengineering

Taplio

Zach Wilson's Linkedin Analytics

Zach Wilson

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts (last 30 days)

Want to drive more opportunities from LinkedIn?

Taplio

.css-1jgreyo{background-image:linear-gradient(284deg, rgba(39,43,72,1) 0%, rgba(33,47,183,1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Zach Wilson's Linkedin Analytics

Zach Wilson

What is Zach talking about?

Who is engaging with Zach

Zach Wilson's Best Posts .css-qqfgvy{font-size:var(--chakra-fontSizes-md);}(last 30 days)

Want to drive more opportunities from LinkedIn?

Zach Wilson's Linkedin Analytics

Zach Wilson's Best Posts (last 30 days)