Unleashing the Potential of Metadata-Driven ELT Framework in Synapse Analytics and Microsoft Fabric

Today, I had the opportunity to speak at the Data Community Austria Day 2024, a full day of sessions dedicated to data enthusiasts and professionals. The event was held at the Jufa Hotel in Wien and attracted more than 300 attendees.

My session was about “Unleashing the Potential of Metadata-Driven ELT Framework in Synapse Analytics and Microsoft Fabric.” I explained how metadata-driven pipelines can help you ingest and transform data with less code, reduced maintenance, and greater scalability than writing code or pipelines for every data source and destination. I also demonstrated how to leverage the medallion architecture (Landingzone, Bronze, Silver, Gold) and Delta tables to build a lakehouse in Microsoft Fabric, a new data platform that combines the best of data lakes and data warehouses.

The audience was very engaged and asked insightful questions. We had a lively discussion about the benefits and challenges of metadata-driven ELT frameworks, the features and capabilities of Synapse Analytics and Microsoft Fabric, and the best practices and tips for implementing them in real-world scenarios.

I want to thank the organizers and sponsors of Data Community Austria Day 2024 for inviting me to speak and for hosting such a wonderful event. I also want to thank all the attendees for their interest and participation. It was a pleasure to share my knowledge and experience with you and to learn from you as well.

My presentation and the code I used can be found on my Github, link below

Data Community Austria 2024