Andrew C. Madson

Tobiko

Head of Education & Evangelism

Wiley

Author

O'Reilly

Apache Polaris - The Definitive Guide
Data Transformation - The Definitive Guide

LinkedIn

✔️Building Data Products with SQL
✔️Hands On Apache Iceberg
✔️Apache Iceberg Fumdamentals
✔️Apache Iceberg for Analytics and ML

LinkedIn Learning Instructor

Insights x Design

Co-Founder

Grand Canyon University

Professor - MSc Business Analytics

Southern New Hampshire University

Professor - Data Analytics

Packt

Tech & Data Advisory Board Member

Trine University

Professor - Masters of Data Science

Dremio

Evangelism - Interim Field CDO

Data Analysis Top Voice | Professor | Product Management @ LPL Financial

I'm a Data & AI Evangelist, Keynote speaker, Opensource Advocate, and Professor.

I LOVE transforming the complex into the accessible. My insights combine my multiple master's degrees in data analytics and business management with hands-on leadership experience at enterprise organizations. 


My technical expertise spans data analytics, machine learning, and artificial intelligence, complemented by strong communication skills developed through roles as an adjunct instructor at six universities and keynote speaker. I bridge the gap between technical innovation and practical application, helping organizations unlock the full potential of their data.

DM to collaborate 🙌

AVP - Product Management - Compliance Analytics and Monitoring

Apache Iceberg separates storage from compute, but which engine should you choose?

Apache Spark, Apache Flink, Dremio, ClickHouse, StarRocks, Snowflake, DuckDB, the list goes on and on!

There are 4 stages of the Iceberg lifecycle -

→ Ingestion
→ Transformation
→ Querying
→ Maintenance

Compartmentalizing each stage of the pipeline helps you choose the right engine.

Here are some other questions to consider:

🧊 Do you need to write to Iceberg from the engine?
→ Spark, Flink, Trino, Dremio, and Athena can write. BigQuery writes only for managed Iceberg, not external.
→ Snowflake writes for both managed Iceberg, and external. ClickHouse is read-only on Iceberg today.

🧊 What is your ops model?
→ Serverless reader on AWS = Athena
→ GCP = BigQuery external or managed
→ Warehouse UX = Snowflake.
→ On-prem or hybrid = Spark, Flink, Trino, Dremio, StarRocks, ClickHouse

🧊 What skills does your team have?
→ SQL-first analytics teams = Trino or Dremio for day-to-day
→ Data engineering teams = Spark and Flink for pipelines + Trino or Dremio for analytics

🧊 Need full platform management for hundreds of sources?
→ Fivetran managed data lake service

Which engine do you use?

#dataengineering #apacheiceberg #data
Casey Karst

Andrew Madson MSc, MBA

DevRel ⚡️O’Reilly Author

October was a busy month for Fivetran OSS projects! 

SQLMesh, SQLGlot, and Connector SDK all saw significant progress 💪

Check out the latest edition of The Monthly Merge for details 😎

Your data lake is a mess. Apache Iceberg can fix it! Here's the mental model that finally made it click for me.

Iceberg is an open table format that organizes your chaotic files into tables. You get ACID transactions, time travel, schema and partition evolution, and the freedom to use multiple engines like Apache Spark, Apache Flink, Dremio, DuckDB and others on the same data.

𝐁𝐮𝐭 𝐡𝐞𝐫𝐞'𝐬 𝐰𝐡𝐚𝐭 𝐦𝐨𝐬𝐭 𝐩𝐞𝐨𝐩𝐥𝐞 𝐦𝐢𝐬𝐬: 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐢𝐬𝐧'𝐭 𝐚𝐧𝐨𝐭𝐡𝐞𝐫 𝐭𝐨𝐨𝐥, 𝐢𝐭'𝐬 𝐚 𝐟𝐮𝐧𝐝𝐚𝐦𝐞𝐧𝐭𝐚𝐥 𝐬𝐡𝐢𝐟𝐭 𝐢𝐧 𝐡𝐨𝐰 𝐰𝐞 𝐭𝐡𝐢𝐧𝐤 𝐚𝐛𝐨𝐮𝐭 𝐝𝐚𝐭𝐚 𝐥𝐚𝐤𝐞 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞.

Iceberg is organized in five levels:

𝟏. 𝐅𝐢𝐥𝐞𝐬
  ⤷ Your Parquet, ORC or Avro data. 
  ⤷ Immutable once committed. 
  ⤷ Column stats are captured for pruning. 

𝟐. 𝐌𝐚𝐧𝐢𝐟𝐞𝐬𝐭𝐬
  ⤷ Lists of data files plus per-file stats and flags. 
  ⤷ Used to prune early and plan efficient scans.

𝟑. 𝐌𝐚𝐧𝐢𝐟𝐞𝐬𝐭 𝐥𝐢𝐬𝐭
  ⤷ The set of manifests that comprises a single snapshot
  ⤷ Includes summary metadata
  ⤷ Each snapshot has exactly one manifest list.

𝟒. 𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 𝐟𝐢𝐥𝐞
  ⤷ Points to the current snapshot ID.
  ⤷ JSON that is the source of truth for the table.
  ⤷ Holds schema, partition and sort orders, properties, history and refs.  

𝟓. 𝐂𝐚𝐭𝐚𝐥𝐨𝐠
  ⤷ Maps table names to the current table metadata file.
  ⤷ Coordinates atomic commits. 
  ⤷ Apache Polaris (Incubating) is the leading OSS option

📖 𝐇𝐨𝐰 𝐫𝐞𝐚𝐝𝐬 𝐰𝐨𝐫𝐤
 Catalog
     ⤷ metadata file
        ⤷ manifest list
           ⤷ manifests
              ⤷ only the files that match your filters.

✍️𝐇𝐨𝐰 𝐰𝐫𝐢𝐭𝐞𝐬 𝐰𝐨𝐫𝐤
New data files 
     ⤷ new manifests
        ⤷ new manifest list
           ⤷ new metadata file
              ⤷ Catalog updates the pointer atomically

S𝑡𝑜𝑝 𝑐ℎ𝑜𝑜𝑠𝑖𝑛𝑔 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑓𝑙𝑒𝑥𝑖𝑏𝑖𝑙𝑖𝑡𝑦 𝑎𝑛𝑑 𝑟𝑒𝑙𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦. With Iceberg, your data lake becomes as trustworthy as your warehouse - while staying completely open.

T𝐡𝐚𝐭'𝐬 𝐰𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞-𝐠𝐫𝐚𝐝𝐞 𝐫𝐞𝐥𝐢𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐨𝐧 𝐨𝐩𝐞𝐧 𝐥𝐚𝐤𝐞 𝐬𝐭𝐨𝐫𝐚𝐠𝐞. 𝐄𝐱𝐚𝐜𝐭𝐥𝐲 𝐰𝐡𝐚𝐭 𝐲𝐨𝐮 𝐧𝐞𝐞𝐝 𝐟𝐨𝐫 𝐀𝐈-𝐫𝐞𝐚𝐝𝐲 𝐝𝐚𝐭𝐚.

#ApacheIceberg #DataEngineering #DataLakehouse

Writing a book isn’t easy. Here's what I learned to get you across the finish line.

The tech landscape shifts fast, it can feel like you’re chasing a moving target. That’s why working with someone you trust is crucial.  

Having a great editorial team to lean on (shoutout O'Reilly) and an amazing co-author (Alex Merced) dramatically reduces stress and significantly improves the final product.  

Grateful for the incredible support that turned a challenging journey into something rewarding - Apache Polaris: The Definitive Guide.

If you’re considering writing or collaborating on tech content, don’t underestimate the power of trusted partners 🙌

If you're in the Bay Area this week, come say hi to me and Alex at:
--> AI By the Bay Conference (Nov 17-19)
--> Fivetran Data+AI Meetup (Nov 18)
--> Apache Polaris (Incubating) Bay Area Meetup (Nov 19)

See you there!

ps. where can I get a Kimberly Incorvia shirt like Alex? Jealous.

Bay Area, let's meet up!

Here's where I'll be:

Nov 15: Theory Ventures Hackathon
🔗 https://lnkd.in/gV7GAkFy
lol - I don't know why I have a UTM...I'm not selling anything

Nov 17-19: AI By the Bay Conference
🔗 https://lnkd.in/gxY2TPSY 
Code BTB654333440 for 25% off
Stoked for Ryan Boyd 🐤, Zhamak Dehghani, and Robert Nishihara

Nov 18: Fivetran Data+AI Meetup
🔗 https://luma.com/l414kx50
The last day to register is November 17 by 12:00 PM PST.
Hear from Elijah D., Alex Merced, and me! 

Nov 19: Apache Polaris (Incubating) Meetup at Snowflake
🔗 https://lnkd.in/eS3CNb8e
Fivetran is providing food, so you know it's going to be good 😎
Hear from Alex Merced, Alvin Zhao, Danica Fine, and me!

Will I see you there?

CelerData Dremio

Apache Iceberg has 5 levels:

LEVEL 1 → Data Files (parquet, avro, orc)
LEVEL 2 → Manifests (column stats, partition values & file paths)
LEVEL 3 → Manifest Lists (all manifests for the snapshot)
LEVEL 4 → Metadata File (schema, partition spec, and snapshot)
LEVEL 5 → Catalog (points to the current metadata file)

In the Bay Area? Come learn about Apache Iceberg and Apache Polaris (Incubating) next week!

Nov 18 - Fivetran User Group
🔗 https://luma.com/l414kx50
🗣️ Elijah D., Alex Merced, and I will do a book giveaway!

Nov 19 - Apache Polaris (Incubating) Meetup at Snowflake 
🔗 https://lnkd.in/eS3CNb8e
🗣️ Alex Merced, Danica Fine, Alvin Zhao, and me!

See you there 🙌

Casey Karst

Model Context Protocol (MCP) may be the most significant shift in data engineering since SQL.

You're managing Fivetran connectors, dbt Labs models, Apache Airflow DAGs, and Snowflake queries. Each tool has its own interface. You're constantly context-switching, looking up API docs, and writing one-off scripts.

Enter MCP 🥳

The open protocol (introduced by Anthropic) lets AI connect directly to your data stack. Instead of navigating UIs or writing API calls, you just... ask.
→ Are all my Fivetran connectors syncing?
→ Show me which dbt tests failed last run
→ Trigger a backfill for yesterday's data

The AI handles the API calls, interprets the results, and gives you answers

Why this matters:
✓ Faster incident response – ask questions in natural language
✓ Lower barrier to entry
✓ Better collaboration
✓ Unified interface

Are you using MCPs?

#DataEngineering #MCP #Data

RIP HIVE METASTORE 🪦

Apache Iceberg REST catalogs, like Apache Polaris (Incubating), are a fundamental shift in how we handle iceberg metadata.

Stateless HTTP services beat stateful RPC protocols every day of the week.

Here's what that unlocks:
→ Horizontal scaling behind load balancers
→ Lock-free concurrency with optimistic control
→ Multi-cloud portability
→ Universal language interoperability
→ Centralized governance

There's a reason why Fivetran uses Iceberg REST catalogs to power our managed data lake service 🙌

Are you using REST?

#DataEngineering #ApacheIceberg #DataLakehouse

To build AI, you don't need to know anything about the underlying business.

That's a perspective that has growing support.

Build an agent, give it access to context, use LLM as Judge, let it loose.

It reminds me of trading versus investing perspectives. 

Trading - all you need to know are patterns and algorithmic trading, nothing about the business.

Investing - You need to know the sector, the industry, the leadership teams, and the financials intimately to make the right decisions.

The answer is probably somewhere in the middle - LLM as Judge paired with a competent human in the loop.

What do you think? How well do you need to know the business to build effective AI?

𝐃𝐚𝐭𝐚 𝐥𝐚𝐤𝐞𝐬 𝐩𝐫𝐨𝐦𝐢𝐬𝐞𝐝 𝐟𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲. 𝐓𝐡𝐞𝐲 𝐝𝐞𝐥𝐢𝐯𝐞𝐫𝐞𝐝 𝐜𝐡𝐚𝐨𝐬.

Schema changes broke queries. Maintenance operations wrecked weekends. Time travel? Good luck with that.

𝐴𝑝𝑎𝑐ℎ𝑒 𝐼𝑐𝑒𝑏𝑒𝑟𝑔 𝑓𝑖𝑛𝑎𝑙𝑙𝑦 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑠 𝑜𝑛 𝑡ℎ𝑒 𝑝𝑟𝑜𝑚𝑖𝑠𝑒 𝑜𝑓 𝑠𝑡𝑜𝑟𝑒 𝑜𝑛𝑐𝑒, 𝑢𝑠𝑒 𝑎𝑛𝑦𝑤ℎ𝑒𝑟𝑒.

𝐇𝐞𝐫𝐞'𝐬 𝐰𝐡𝐲 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐦𝐚𝐭𝐭𝐞𝐫𝐬 🧊

𝐒𝐜𝐡𝐞𝐦𝐚 𝐞𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧
→ Add, drop, rename columns without rewriting data
→ Your existing queries keep running
→ No more broken pipelines from schema changes

T𝐢𝐦𝐞 𝐭𝐫𝐚𝐯𝐞𝐥
→ Every write creates a snapshot
→ Roll back bad data in seconds
→ Debug by comparing table states over time
→ Reproduce last month's analysis exactly

H𝐢𝐝𝐝𝐞𝐧 𝐩𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐢𝐧𝐠
→ Writers don't need to know partition schemes
→ Readers don't care about partition paths
→ Update strategies without breaking anything
→ Zero data rewrites needed

A𝐂𝐈𝐃 𝐭𝐫𝐚𝐧𝐬𝐚𝐜𝐭𝐢𝐨𝐧𝐬
→ Multiple writers modify tables safely
→ No corrupted tables from failed jobs
→ Atomic commits or nothing
→ Reliability that matches warehouses

The momentum is strong 🚀

Snowflake, Databricks, Amazon Web Services (AWS), Dremio, and Fivetran all actively support Iceberg.

Yo𝐮 𝐧𝐨 𝐥𝐨𝐧𝐠𝐞𝐫 𝐡𝐚𝐯𝐞 𝐭𝐨 𝐜𝐡𝐨𝐨𝐬𝐞 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐟𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐫𝐞𝐥𝐢𝐚𝐛𝐢𝐥𝐢𝐭𝐲. 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐠𝐢𝐯𝐞𝐬 𝐲𝐨𝐮 𝐛𝐨𝐭𝐡.

Ar𝑒 𝑦𝑜𝑢 𝑢𝑠𝑖𝑛𝑔 𝐼𝑐𝑒𝑏𝑒𝑟𝑔?

➡️ Shoutout to Danica Fine for inviting me to speak at the Apache Polaris (Incubating) meetup last night. Excited to be back in the Iceberg/Polaris community 🙌

#ApacheIceberg #DataEngineering #DataLakehouse

Shoutout to the winners of Theory Ventures America's Next Top Modeler 👏

Congrats to Bryan Bischof and team for putting together an amazing event. 

It was great to see so many people come out on a Saturday and build with AI. 

Looking forward to the next one 😎

taplio

Andrew Madson MSc, MBA's Linkedin Analytics

Andrew C. Madson

Performance ScoreAndrew C. Madson's LinkedIn Performance

What is Andrew talking about?

Who is engaging with Andrew

Andrew C. Madson's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Andrew Madson MSc, MBA's Linkedin Analytics

Andrew C. Madson

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Andrew C. Madson's LinkedIn Performance

What is Andrew talking about?

Who is engaging with Andrew

Andrew C. Madson's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Andrew Madson MSc, MBA's Linkedin Analytics

Performance ScoreAndrew C. Madson's LinkedIn Performance

Andrew C. Madson's Best Posts (last 30 days)