Learn Data Engineering

The ultimate Data Engineering Academy. Teaching you everything you need to become a Data Engineer or add Data Engineering to your skillset. Proven system through my 10+ years of experience, over 2,000 students, over 50hours of video training material.

Founder

Udemy

I create free data & AI courses on Udemy for Data Engineers, Analysts, Scientists and beginners.

Course Author & Instructor

Kretz Media

At Kretz Media, we help innovative companies share their story and connect with the right audience. We specialize in creating authentic, high-impact content that makes complex topics understandable – and your brand unforgettable.

Whether it's a YouTube product demo, a livestream, a branded online course, or a strategic LinkedIn campaign – we turn your innovation into media that resonates.

LinkedIn

In this course you learn everything you need to know to design a NoSQL graph database and implement it in the cloud.

LinkedIn Learning Instructor

Bosch Rexroth

Data Lab Team Lead

Data Engineering Team Lead

Data Engineer

SALT Solutions GmbH

SAP Consultant

Hey there! I'm Andreas Kretz, your dedicated guide to the exciting realm of Data Engineering and the founder of the Learn Data Engineering Academy. With over years of specialized coaching experience, I've had the pleasure of helping more than 2,000 students reach their goals in this essential field.

I bring over 8 years of experience in Corporate Data Science, 14+ years in Computer Science and IoT, and a robust background as a Data Engineering and Data Labs Team Lead. My journey in Data Engineering has garnered global recognition, including being named a LinkedIn Top Voice in Data Science & Analytics in 2018, 2019, and 2024.

Here's what I do:
➡️ Learn Data Engineering Academy
I’ve trained over 2,000 students through my Academy, offering 30+ in-depth courses and projects with over 55 hours of video content, covering everything you need to excel in Data Engineering.

➡️ Media Work
In 2024, I helped my media partners achieve significant brand awareness, resulting in 1.21 million post views on LinkedIn, 250,000 video views on YouTube, and 235,000 sent emails with a 40% open rate. My efforts have built a vast network of over 100,000 engineers and scientists.

➡️ YouTube
On my YouTube channel, I simplify complex Data Engineering concepts and share valuable insights from the Academy. Join me for livestreams with industry experts, workshops, tool reviews, and deep dives into the latest architectures. It’s your go-to source for staying ahead in the data world.

If you're passionate about Data Engineering, eager to learn, or curious about the latest tools and techniques, hit subscribe. Let's explore the world of Data Engineering together!

Are pure vector stores going to die? 
For the past few years everyone was hyped about vector stores. Me too ;) but companies actually start implementing vector store features (spoiler: and more) into their existing databases.

I've been looking into Oracle's 26ai database and it's a prime example why pure vector stores are in BIG trouble. Because Oracle is not just implementing the vector store and search feature, but enables you to build full RAG pipelines directly on the database.

Let me explain the typical process first, so you understand why I think this is a big deal. 

With a vector store:
1️⃣ You have your data in a data store (let's say MongoDB)
2️⃣ You need to use a Python script to copy the data over to the vector store
3️⃣ In that script you also need to create the embeddings and store the vectors
4️⃣ Then you query the vector store with your question 
5️⃣ And then you send the result to an LLM to get your polished answer

With 26ai:
1️⃣ You upload hugging face transformers to the database and create a PL/SQL transformation script (this is only done once and needs a bit of setup. More on that later)
2️⃣ After the setup you store your data in the relational db (also takes JSONs btw) and embeddings are being created automatically in a vector column. 
3️⃣ For queries you can build a full RAG pipeline with PL/SQL, because the db can directly connect to external LLMs (local or cloud) through API calls. This way you'll get the answer directly without you needing to feed something to the LLM.

→ So, not just vector storage and search, but full RAG pipelines.
And if you have JSONs, the db can automatically infer a relational schema!

I really like the idea of having everything, the embedding logic, semantic search, and RAG directly in the database. Set it up once and anyone who's using the db just uses it like they always do. 

Now, one thing I don't like that much:
To set up automatic embedding you first need to download your hugging face transformer like all-MiniLM-L6-v2, turn it into an ONNX model with OML4Py 2.0 and upload it to the db. ONNX (Open Neural Network Exchange) is an open, standardized file format for running machine-learning models across different platforms. (research ONNX!)

Personally, I don’t like these kinds of workflows where you need to download something, then install tools to convert X to Y and upload Y somewhere. Why can't I just upload the transformer to the db and it does the rest? Or tell the db "I want to use this hugging face transformer" and it does everything for me?

I would love to see that simplification.

I'm absolutely fine with the PL/SQL pipeline creation parts by the way. Your data and your use case is always custom and you need a way to customize these pipelines.

So who needs a dedicated vector store again?

You can try out the 26ai database for FREE right here: ﻿https://fandf.co/4pp6YxR﻿
 
*This post is sponsored by Oracle*
 
#DataEngineering #AI #VectorSearch #Oracle26ai #RAG #ModernDataStack

Andreas Kretz

Follow for daily Data Engineering content!
Teaching you Data Engineering at my Academy and helping companies recruit top engineering talent.
Huge network of over 100,000 engineers & scientists + private engineering network of over 1000 students through my Data Engineering Academy

#dataengineering #dataengineer #datascience #bigdata

I teach Data Engineering and create data & AI content | 10+ years of experience | 3x LinkedIn Top Voice | 230k+ YouTube subscribers

Angelo  viloria 

I just tested something really cool: I ran a SQL query that combined data from Snowflake and BigQuery without any data movement!
 
I used Zetaris to connect both systems as virtual sources, created a data mart, and ran federated queries like everything was in one place. No ETL, no duplication.
 
Honestly, I think this could be a real alternative to the lakehouse setup.
 
I recorded the whole process for YouTube.
🎥 If you're working with data across systems, check it out: ﻿
https://lnkd.in/eNbRf6TU
 
Let me know in the comments what you think about this approach. I’d love to hear your take!
 
#DataEngineering #SQL #BigQuery #Snowflake #Zetaris #Sponsored

Zetaris

Shubham Upadhyaya

Tools are cool. Systems are smarter.
 
 
A few days ago, one of my students messaged me super excited:
“Hey Andreas, I just started a new job. I’m working with Airflow, Redshift, Grafana, and a few other tools.”

That’s a solid stack. Really good tools. So I said:
“Awesome! And how do they all connect together?”

...And then: silence.

No answer. Not because the student didn’t want to tell me, but because they hadn’t thought about it. They didn’t know.

And that’s exactly the problem I see all the time.

People are learning tools in isolation.
They know how to click through Grafana, create a DAG in Airflow, or write a query in Redshift, but they don’t see the system behind it.

Here’s the truth:
Nobody is hiring you to just use use a single tool.
They’re hiring you to solve a business problem.
And Airflow or Redshift are just one piece in the puzzle.

If you only focus on tools, you’ll always be stuck thinking in checkboxes:
✅ Airflow
✅ Redshift
✅ Grafana

But that’s not what makes a good engineer.

What makes a good engineer is the ability to design systems that work.
That deliver value. That solve real-world problems.

So the mindset has to shift from “Which tool should I learn next?” to “How does the full system actually work?”

Here are a few questions I always ask my students:

👉 Where does the data come from?
Is it batch? Is it streaming? From APIs, databases, or message queues?
👉 How does it flow through the system?
What are the processing steps? Is there a transformation layer? Is it orchestrated with Airflow, or something else?
👉 Where does the data get stored?
Is it Redshift? A data lake? Both?
👉 Who consumes the data in the end?
Are there dashboards? Machine learning models? Business users doing ad-hoc analysis?
👉 What happens when something fails?
Is there monitoring? Logging? Alerting? Can someone find the root cause?

Once you can answer these kinds of questions, you’re no longer just “someone who knows a few tools.”

You’re someone who can build and maintain real systems.
You’re thinking like an engineer.

And yes, tools are still important. You need to know how they work.
But without system thinking, you're just reacting. You’re not designing.

So: next time you learn a new tool, ask yourself: how does this fit into the bigger picture?

Tools come and go. But the ability to understand and design systems?
That’s what will really move your career forward.

🚨 Hey everyone, there's a free Astronomer webinar coming up that shows you how to implement proper CI/CD strategies for your Airflow pipelines! 
 
If you're working with Airflow and still pushing DAGs manually or copying files over… it's time for an upgrade.
 
One thing I always tell my students: treat your pipelines like software.
Because that's what they are, basically.
 
You write code, you version it, you test it (hopefully), and you deploy it. And if you're still doing all that manually, you're going to hit a wall sooner or later.
 
Astronomer's Marc Lamberti is diving into this topic in an upcoming webinar:
𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗶𝗻𝗴 𝗖𝗜/𝗖𝗗 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝗳𝗼𝗿 𝘆𝗼𝘂𝗿 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀
🗓 Thursday, December 4 | 11am ET
 
👉 Save your spot now:
https://lnkd.in/eb8-SgC9
 
Pro tip: register even if you can't make it. You'll receive the recording afterwards. 
 
You’ll learn:
✅ How CI/CD actually works with Airflow
✅ How to use Astro’s GitHub integration and other CI/CD features
✅ Strategies for deploying your DAGs the right way
 
If you want your Airflow projects to grow with your team and not break every time someone pushes a change, don’t skip this one 😉

🔥30% BLACK FRIDAY DEAL!!🔥
 
I just set up a 30% discount for the 12 months and unlimited access to my Data Engineering Academy.
 
⚠️ Use the code BLACKFRIDAY25 at checkout and join 2,000+ students now:
https://lnkd.in/e59HGcJu

This deal expires December 1st 11:59 PM ⏰
 
You get:
✅ Huge Academy with 30+ hands-on courses and projects 
✅ Built by a real Data Engineer with 10+ years of experience
✅ AI Engineering Track that teaches you to build real AI solutions
✅ Tools & platforms that companies actually hire for
✅ Clear roadmaps for every background
✅ Private Discord community with over 2,000 members
✅ Constant updates with new courses & projects
 
🎁 FREE Black Friday Bonus:
My full Academy course “Embeddings: The Hidden Foundation of GenAI” is available in the Free Labs & Partners section until December 1st! The perfect intro to our AI Engineering Track.
 
Get the free labs: https://lnkd.in/eANytAfd
 
#dataengineering #learndataengineering #aiengineering

taplio

Andreas Kretz's Linkedin Analytics

Andreas Kretz

Performance ScoreAndreas Kretz's LinkedIn Performance

What is Andreas talking about?

Who is engaging with Andreas

Andreas Kretz's Best Posts (last 30 days)

Ready to grow your LinkedIn presence?

taplio

.css-6ytdur{background-image:linear-gradient(284deg, rgba(39, 43, 72, 1) 0%, rgba(33, 47, 183, 1) 100%);color:transparent;-webkit-background-clip:text;background-clip:text;}Andreas Kretz's Linkedin Analytics

Andreas Kretz

Performance Score.css-goub6s{font-size:var(--chakra-fontSizes-md);font-weight:500;color:var(--chakra-colors-gray-600);display:block;margin-top:var(--chakra-space-1);}Andreas Kretz's LinkedIn Performance

What is Andreas talking about?

Who is engaging with Andreas

Andreas Kretz's Best Posts .css-1emoo84{font-size:var(--chakra-fontSizes-md);font-weight:400;color:var(--chakra-colors-gray-600);}(last 30 days)

Ready to grow your LinkedIn presence?

Andreas Kretz's Linkedin Analytics

Performance ScoreAndreas Kretz's LinkedIn Performance

Andreas Kretz's Best Posts (last 30 days)