Erwin | Data & Intelligence
  • Home
  • FMD Framework
  • Speaker Profile
  • About
  • Contact
Select Page
Fabric Metadata Driven Framework November 2026 Release Notes

Fabric Metadata Driven Framework November 2026 Release Notes

by Erwin | Nov 15, 2026 | FMD Framework

Fabric Metadata‑Driven Framework (FMD)

Lakehouse

by Erwin | Nov 15, 2026

Table of Contents

No headings found in this post.

    Microsoft Fabric Metadata Driven Framework November 2026 Release

    One of the most common questions I get about the Fabric Metadata-Driven Framework (FMD) is: "How can I load some tables every hour and the rest only once a day?"

    Schedule Groups in the FMD Framework: load the right entities at the right time

    Until now, PL_FMD_LOAD_LANDINGZONE picked up every active landing-zone entity in the workspace. That works well when everything runs on the same cadence. In practice, though, sales orders may need to be refreshed hourly while reference data only changes overnight.

    Another frequent request from the community was to load entities by source system or by department. When several teams or sources share one FMD deployment, you want to run each one independently, so the solution can scale out without one large load that does everything.

    The new Schedule Groups feature solves both. You keep all metadata in the configuration database, group entities by schedule, source, or department, and choose which group to run when you start the pipeline. You can now create several schedules on the PL_FMD_LOAD_ALL pipeline, each with its own ScheduleGroupName parameter value.

    What is a Schedule Group?

    A schedule group is a named set of landing-zone entities, such as Hourly, Nightly, or Finance. When you start a pipeline, you pass the name of the group, and only the entities in that group are loaded.

    A few design principles:

    • An entity can belong to multiple groups. For example, a table can be in both Hourly and Finance. It appears only once in each group.
    • The Default group always exists. Every landing-zone entity is a member, so existing deployments keep working without changes.
    • The schedules themselves still live in Fabric. A schedule group decides what is loaded. A Fabric pipeline schedule decides when.

    What changed?

    1. A new ScheduleGroupName pipeline parameter

    Every Fabric Data Pipeline in the framework, except PL_FMD_LOAD_BRONZE and PL_FMD_LOAD_SILVER, now has a string parameter named ScheduleGroupName that defaults to Default.

    Each pipeline passes the value unchanged to the Fabric pipelines it invokes. For example, if you start PL_FMD_LOAD_ALL with ScheduleGroupName = Hourly, every landing-zone command and copy pipeline in the chain receives Hourly.

    Bronze and Silver are not filtered by schedule group. They continue to process whatever has landed and been queued.

    2. Two new tables in the configuration database

    Table Purpose
    integration.ScheduleGroup One row per group: ScheduleGroupId and a unique ScheduleGroupName
    integration.ScheduleGroupEntity The relationship between a schedule group and a landing-zone entity. (ScheduleGroupId, LandingzoneEntityId) is the primary key.

    3. The landing-zone view now includes the group

    execution.vw_LoadSourceToLandingzone now joins the schedule-group tables and exposes ScheduleGroupId and ScheduleGroupName. Every lookup that reads this view, 21 in total, now filters by the pipeline parameter:

    ... AND ScheduleGroupName = N'@{replace(pipeline().parameters.ScheduleGroupName, '''', '''''')}'

    Apostrophes in group names are escaped, so a name such as Erwin's tables works correctly.

    Backward compatibility: the Default group

    I didn't want existing deployments to require any manual work. That is handled in three ways:

    1. On deployment, the DACPAC post-deployment script runs integration.sp_InitializeDefaultScheduleGroup. It creates the Default group if it doesn't exist and adds all existing landing-zone entities, including inactive ones. It is idempotent and never removes custom memberships.
    2. On every new insert, the trigger integration.tr_LandingzoneEntity_DefaultScheduleGroup adds the new entity to Default. This applies whether the entity is created through sp_UpsertLandingzoneEntity, sp_UpsertLandingzoneBronzeSilver, a direct INSERT, or a multi-row insert. The trigger runs in the same transaction, so rolling back the insert also rolls back the membership.
    3. Pipeline parameters default to Default, so schedules that don't pass the parameter continue to load the same active entities as before.

    An unknown or empty group name returns no entities; it does not fall back to loading everything. If you have saved runs or schedules that explicitly pass an empty string, change the value to Default.

    Managing groups: two new stored procedures

    Create or rename a group

    -- Create the group, or return the existing one
    EXECUTE [integration].[sp_UpsertScheduleGroup] @ScheduleGroupName = N'Hourly';
    
    -- Rename a group by its id
    EXECUTE [integration].[sp_UpsertScheduleGroup]
        @ScheduleGroupId   = 2,
        @ScheduleGroupName = N'EveryHour';

    The procedure is idempotent and returns the group row. You cannot rename the Default group, because the insert trigger relies on that name.

    Add an entity to a group

    EXECUTE [integration].[sp_UpsertScheduleGroupEntity]
        @LandingzoneEntityId = 42,
        @ScheduleGroupName   = N'Hourly';

    The combination of landing-zone entity and schedule group is unique, so running the procedure again doesn't create a duplicate. It fails with a clear error if the entity or the group doesn't exist.

    To add a whole data source at once:

    DECLARE @Id BIGINT;
    DECLARE c CURSOR LOCAL FAST_FORWARD FOR
        SELECT LandingzoneEntityId
        FROM integration.LandingzoneEntity
        WHERE DataSourceId = 3;
    OPEN c;
    FETCH NEXT FROM c INTO @Id;
    WHILE @@FETCH_STATUS = 0
    BEGIN
        EXECUTE integration.sp_UpsertScheduleGroupEntity
            @LandingzoneEntityId = @Id,
            @ScheduleGroupName   = N'Hourly';
        FETCH NEXT FROM c INTO @Id;
    END;
    CLOSE c;
    DEALLOCATE c;

    Putting it together: hourly and nightly loads

    1. Create the groups:
      EXEC integration.sp_UpsertScheduleGroup @ScheduleGroupName = N'Hourly';
    2. Add the fast-changing entities to Hourly with sp_UpsertScheduleGroupEntity.
    3. In Fabric, add two schedules to PL_FMD_LOAD_ALL:
      • Every hour, with ScheduleGroupName = Hourly
      • Every night, with ScheduleGroupName = Default, which still contains everything
    4. Check which entities a run will pick up:
      SELECT EntityId, SourceName, ScheduleGroupName
      FROM execution.vw_LoadSourceToLandingzone
      WHERE ScheduleGroupName = N'Hourly';

    Because every entity also remains in Default, the nightly run acts as a full safety net, while the hourly run keeps your critical tables fresh.

    Scaling out: one schedule per source or department

    The same approach works for organizing loads by source system or department, instead of only by time. Create a group for each one, for example SAP, Salesforce, Finance, and HR, and add the relevant entities to it.

    Then, in Fabric, open PL_FMD_LOAD_ALL → Schedule and add one schedule per group. In each schedule, set the ScheduleGroupName parameter to the group:

    Schedule Recurrence ScheduleGroupName
    SAP load Every hour SAP
    Salesforce load Every 4 hours Salesforce
    Finance load Daily at 06:00 Finance
    HR load Daily at 02:00 HR
    Full safety net Nightly Default

    Each schedule runs the same pipeline chain but loads only its own entities. This brings several benefits:

    • Independent runs. A failing source doesn't block the other sources or departments.
    • Easy onboarding. Adding a new source or department means creating a group and a schedule, not a new pipeline.
    • Better capacity planning. You can spread loads across the day instead of running one large batch.
    • Clear ownership. Each department can see and manage its own group of entities.

    Upgrading

    Redeploy the framework database and pipelines together using NB_SETUP_FMD.ipynb. Deploy both at the same time: the new pipelines require the new view columns, and the old pipelines don't pass a schedule group. The DACPAC creates the tables, updates the view, adds the procedures and trigger, and initializes the Default group automatically. No manual SQL is required.

    See the deployment guide for details.

    Wrap-up

    Schedule Groups let you run different load cadences from the same metadata-driven pipelines, without duplicating pipelines or adding custom filtering. Existing deployments keep working because of the Default group, and new groups take just two stored-procedure calls.

    As always, feedback, issues, and pull requests are welcome on GitHub. Find the full documentation at erwindekreuk.com/fmd-framework.

    New to the Microsoft Fabric Metadata Driven Framework?

    Learn more about the framework architecture, deployment model, and features here:

    FMD Framework – Metadata-Driven Data Integration for Microsoft Fabric

    Stay Connected

    The Microsoft Fabric Metadata Driven Framework continues to help organizations standardize data platform delivery, automate Microsoft Fabric deployments, improve governance, and accelerate AI-ready architectures. With every release, the framework becomes more reliable, secure, and easier to adopt for enterprise-scale Microsoft Fabric implementations.

     

    GitHub Repository
    FMD_FRAMEWORK

    Hit Subscribe to get future release notes delivered straight to your inbox!

    Latest FMD Releases

    • Fabric Metadata‑Driven Framework (FMD)
      Fabric Metadata Driven Framework November 2026 Release Notes
    • Fabric Metadata‑Driven Framework (FMD)
      Fabric Metadata Driven Framework July 2026 Release Notes
    • Fabric Metadata‑Driven Framework (FMD)
      Bringing Column-Level Lineage from Microsoft Fabric to Microsoft Purview with FMD
    • Fabric Metadata‑Driven Framework (FMD)
      Fabric Metadata Driven Framework update May 2026
    • Fabric Metadata‑Driven Framework (FMD)
      FMD Framework Update April 2026: Identity Improvements, SCD2 Fixes & Stability Enhancements

    Latest Posts

    • Fabric Metadata‑Driven Framework (FMD)
      Fabric Metadata Driven Framework November 2026 Release Notes
    • Fabric Metadata‑Driven Framework (FMD)
      Fabric Metadata Driven Framework July 2026 Release Notes
    • Fabric Logo
      Reassigning Microsoft Fabric Capacities in Bulk: Don’t Let an Expiring Trial Catch You by Surprise
    • Fabric Metadata‑Driven Framework (FMD)
      Bringing Column-Level Lineage from Microsoft Fabric to Microsoft Purview with FMD
    • Rayfin
      My first experience: Building a Fabric App

    Categories

    • Azure (49)
    • Azure Databricks (9)
    • Blogs (1)
    • Divers (9)
    • Domoticz (1)
    • Events (38)
    • FMD Framework (12)
    • Microsoft Fabric (16)
    • Microsoft Fabric Content HUB (9)
    • Microsoft Purview (19)
    • Power BI (1)
    • SQL Server (3)
    • Sql Server Management Studio (2)

    Feel free to leave a comment

    Fabric Metadata Driven Framework November 2026 Release Notes

    Fabric Metadata Driven Framework July 2026 Release Notes

    by Erwin | Jul 16, 2026 | FMD Framework

    Fabric Metadata‑Driven Framework (FMD)

    Lakehouse

    by Erwin | Jul 16, 2026

    Table of Contents

    No headings found in this post.

      Microsoft Fabric Metadata Driven Framework July 2026 Release

      The Microsoft Fabric Metadata Driven Framework (FMD Framework) continues to evolve rapidly, driven by real-world customer implementations and an amazing open-source community. This July 2026 release focuses on improved lineage and governance, simplified deployment automation, stronger reliability, and enhanced security across the framework.

      Before diving into the details, I would like to give special recognition to one contributor in particular.

      Special Thanks to Matthias Falland

      This release would not have been possible without the incredible contributions from Matthias 

      A large number of improvements in this release from deployment automation and security hardening to pipeline reliability and lineage enhancements, were contributed, reviewed and refined through their efforts.

      Open-source projects thrive because of people who are willing to invest their expertise and time to make the platform better for everyone. The quality, attention to detail, and commitment to enterprise-grade reliability demonstrated in these contributions have significantly strengthened the FMD Framework.

      A sincere thank you for helping make FMD more robust, secure, and adoption-ready for organizations building modern data platforms on Microsoft Fabric.

      What's New in the Microsoft Fabric Metadata Driven Framework July 2026 Release

      Enhanced Purview Lineage Integration

      Following the recent introduction of the Fabric-to-Purview Lineage Extractor, several improvements have been implemented to simplify lineage management and make deployments more reliable.

      Folder-Based Lineage Processing

      One of the most requested enhancements was changing lineage extraction from a file-based approach to a folder-based approach.

      Benefits:

      • Simplified landing zone configuration
      • Reduced maintenance effort
      • More logical alignment with real-world data lake architectures
      • Improved support for complex ingestion scenarios

      This change allows FMD to automatically process all relevant files within a designated folder instead of requiring explicit file references.

      Improved Workspace Reference Handling

      The lineage extractor now correctly references Fabric Workspace identifiers through deployment placeholders, making multi-environment deployments considerably simpler.

      Benefits:

      • Easier Dev/Test/Prod deployments
      • Reduced manual configuration
      • Improved portability between tenants

      Simplified Deployment and Onboarding

      A major focus of this release was reducing deployment complexity.

      Automated Fabric SQL Connection Creation

      The Microsoft Fabric Metadata Driven Framework now supports automated Fabric SQL connection creation. You can now automatically create the CON_FMD_FABRIC_SQL connection using a Service Principal.

      Why this matters

      Previously, administrators needed to manually create and configure the connection before deployment.

      Now FMD can:

      • Create the connection automatically
      • Configure authentication
      • Reduce post-deployment manual steps
      • Accelerate onboarding

      This significantly improves first-time installations.

      Improved Key Vault Integration

      A new configuration option enables SQL connections to leverage Azure Key Vault.

      Benefits

      • Centralized credential management
      • Better security posture
      • Easier credential rotation
      • Reduced operational risk

      This aligns FMD more closely with enterprise security and governance standards.

      Reliability Improvements Across Data Pipelines

      This release contains several reliability-focused enhancements that improve monitoring, troubleshooting, and operational transparency.

      Better Failure Detection

      Multiple pipelines now correctly report failures instead of reporting successful execution when an underlying process fails.

      Enhancements include:

      • PL_FMD_LOAD_ALL
      • Landing Zone processing
      • Landing Zone audit activities
      • Pipeline logging improvements

      Operational impact

      Data engineers and platform teams can now:

      • Detect issues faster
      • Trust audit results
      • Improve monitoring accuracy
      • Reduce troubleshooting time

      Improved Pipeline Correlation Tracking

      Pipeline executions now receive improved PipelineParentRunGuid fallback logic.

      Extended support was added for:

      • Notebook executions
      • Bronze processing
      • Silver processing

      This improves observability and end-to-end execution tracking.

      Watermark Synchronization Improvements

      An issue was resolved where watermark advancement could occur before queue insertion completed.

      The process is now serialized correctly to ensure:

      • Better incremental load reliability
      • Reduced chance of missed records
      • Improved consistency during high-volume processing

      Data Quality and Transformation Improvements

      Several improvements have been made to metadata-driven transformations.

      Improved Hash Generation

      Stable Primary Key Hashes

      Hashed primary keys are now generated independently of source column order.

      Benefits:

      • Consistent hash generation
      • Reduced risk during schema changes
      • Improved reproducibility

      Better Change Detection

      Primary key columns are now excluded from HashedNonKeyColumns calculations.

      This improves:

      • Change tracking accuracy
      • Delta detection reliability
      • Incremental processing performance

      DACPAC Validation

      Build pipelines now verify that:

      • DACPAC artifacts match the SQL source project
      • SQL changes are correctly included
      • Deployment packages remain synchronized

      Additionally:

      • DACPAC generation has been automated
      • Deployment consistency has improved
      • Release quality has increased

      Additional Fixes

      This release also includes numerous bug fixes and quality-of-life improvements:

      • Fixed StartCopyActivity typo affecting ADLS and ADF visibility
      • Added missing invoke_fabric_api_request implementation
      • Corrected deployment guide sequencing
      • Improved demo data deployment
      • Enhanced OneLake configuration deployment
      • Updated Lakehouse assets and DACPAC packaging
      • Improved namespace handling to prevent truncation issues

      Community Contributions

      A special thank you goes to Matthias Falland  for a significant number of high-quality contributions in this release. These improvements help make FMD more reliable, secure, and easier to deploy across enterprise environments.

      Open-source collaboration continues to play a crucial role in making FMD the leading metadata-driven framework for Microsoft Fabric.

      Conclusion

      This July release is less about introducing flashy new features and more about something equally important: enterprise readiness.

      The improvements delivered in this update strengthen the framework in four critical areas:

      ✅ Better Purview integration and lineage management
      ✅ Simplified deployments and onboarding
      ✅ Improved operational reliability and monitoring
      ✅ Stronger security and governance controls

      For organizations building large-scale Microsoft Fabric implementations, these enhancements further reduce deployment complexity while increasing trust in metadata-driven automation.

      The FMD Framework continues its mission of helping organizations build scalable, governed, and AI-ready data platforms on Microsoft Fabric.

      New to the Microsoft Fabric Metadata Driven Framework?

       Learn more about the framework architecture, deployment model, and features here:

      FMD Framework – Metadata-Driven Data Integration for Microsoft Fabric

      Stay Connected

       

      The Microsoft Fabric Metadata Driven Framework continues to help organizations standardize data platform delivery, automate Microsoft Fabric deployments, improve governance, and accelerate AI-ready architectures. With every release, the framework becomes more reliable, secure, and easier to adopt for enterprise-scale Microsoft Fabric implementations.

       

      GitHub Repository
      👉FMD_FRAMEWORK

      Hit Subscribe to get future release notes delivered straight to your inbox!

      Latest FMD Releases

      • Fabric Metadata‑Driven Framework (FMD)
        Fabric Metadata Driven Framework November 2026 Release Notes
      • Fabric Metadata‑Driven Framework (FMD)
        Fabric Metadata Driven Framework July 2026 Release Notes
      • Fabric Metadata‑Driven Framework (FMD)
        Bringing Column-Level Lineage from Microsoft Fabric to Microsoft Purview with FMD
      • Fabric Metadata‑Driven Framework (FMD)
        Fabric Metadata Driven Framework update May 2026
      • Fabric Metadata‑Driven Framework (FMD)
        FMD Framework Update April 2026: Identity Improvements, SCD2 Fixes & Stability Enhancements

      Latest Posts

      • Fabric Metadata‑Driven Framework (FMD)
        Fabric Metadata Driven Framework November 2026 Release Notes
      • Fabric Metadata‑Driven Framework (FMD)
        Fabric Metadata Driven Framework July 2026 Release Notes
      • Fabric Logo
        Reassigning Microsoft Fabric Capacities in Bulk: Don’t Let an Expiring Trial Catch You by Surprise
      • Fabric Metadata‑Driven Framework (FMD)
        Bringing Column-Level Lineage from Microsoft Fabric to Microsoft Purview with FMD
      • Rayfin
        My first experience: Building a Fabric App

      Categories

      • Azure (49)
      • Azure Databricks (9)
      • Blogs (1)
      • Divers (9)
      • Domoticz (1)
      • Events (38)
      • FMD Framework (12)
      • Microsoft Fabric (16)
      • Microsoft Fabric Content HUB (9)
      • Microsoft Purview (19)
      • Power BI (1)
      • SQL Server (3)
      • Sql Server Management Studio (2)

      Feel free to leave a comment

      Categories

      • Azure (49)
      • Azure Databricks (9)
      • Blogs (1)
      • Divers (9)
      • Domoticz (1)
      • Events (38)
      • FMD Framework (12)
      • Microsoft Fabric (16)
      • Microsoft Fabric Content HUB (9)
      • Microsoft Purview (19)
      • Power BI (1)
      • SQL Server (3)
      • Sql Server Management Studio (2)
      Adding an RSS feed to this site’s homepage is not supported, as it could lead to a loop that slows down your site. Try using another block, like the Latest Posts block, to list posts from the site.
      • RSS

      Designed by Elegant Themes | Powered by WordPress