Fabric Metadata-Driven Framework (FMD): Simplifying Enterprise Data Integration

Architecture at a Glance

FMD implements the Medallion Lakehouse pattern in Fabric with Delta as the canonical table format.

Layers

Landing Zone: file landing and raw ingestion into Lakehouse files/tables.
Bronze: deduplication/structural normalization into Delta tables.
Silver: historical tracking, validations, and SCD‑oriented patterns.
Gold: (Business Domains) semantic/BI models if your solution scope requires it.

Workspaces & Separation of Concerns

Configuration: central Fabric SQL DB with metadata model, logging, and audit tables.
Data: Landing, Bronze, Silver Lakehouses per environment.
Code: Pipelines, Notebooks, and Spark artifacts per environment.

Business Domains & Gold Layer

FMD enables domain‑driven design for analytics:

Domain Separation: Each business domain (e.g., Sales, HR, Finance) gets its own data workspace (Gold) and code workspace (pipelines, notebooks).
Gold Layer Examples: Delivered templates show how to:

• Create dimension tables (e.g., DimCustomer, DimEmployee).
• Build fact tables (e.g., FactSales, FactPayroll).

Customizable: Extend templates for additional domains.

Metadata Model (Core Tables)

FMD’s configuration lives in a Fabric SQL Database (Integration + Execution schemas). Below are representative entities: [github.com]

[integration].Connection – Fabric connections (GUID, type, gateway type, datasource reference).
[integration].DataSource – systems/domains (name, namespace/prefix, type such as ASQL_01, ADLS_01, ADF, ONELAKE_*).
[integration].LandingzoneEntity – source schema/name, file type/path, incremental flags/columns.
[integration].BronzeLayerEntity – target schema/name, PK(s), cleansing rules, Delta as default file type.
[integration].SilverLayerEntity – downstream table metadata and cleansing rules.
[execution].LandingzoneEntityLastLoadValue & [execution].PipelineLandingzoneEntity – last‑load bookmarks and pipeline/file processing ledger.

Stored procedures (e.g., sp_UpsertLandingzoneEntity, sp_UpsertBronzeLayerEntity, sp_UpsertLandingZoneEntityLastLoadValue) are included for reliable metadata mutation and incremental bookkeeping.
More details can be found in WIKI

Orchestration  & Processing Patterns

Orchestrator Pipelines resolve what to run via metadata lookups and call child pipelines / notebooks for Landing → Bronze → Silver progression.
Supported Activities include Lookup, Copy, Get Metadata, Delete, Script, Stored Procedure—standardizing execution across sources.
Bulk Onboarding: PL_TOOLING_POST_ASQL_TO_FMD extracts table metadata from SQL sources and populates FMD metadata tables

Logging & Observability

Out‑of‑the‑box audit tables capture execution across pipeline runs, notebooks, and Copy activities:

audit.PipelineExecution (start/end/fail per pipeline run).
audit.CopyActivityExecution (start/end/fail per copy activity).
audit.NotebookExecution (start/end per notebook).

You can wire these tables into your monitoring solution or build custom Fabric semantic models on top.

Ready to accelerate your ingestion & standardize your pipelines?

To deploy the framework:

1. Review the Deployment Guide – Walks through prerequisites and setup steps, can be found in the GitHub repository.

Video’s

Video with Patrick LeBlanc @Fabric Tech Talk Fridays

Video with Matthias Falland -The Trusted Advisor

Video with the Data Factory Team