Fabric Metadata Driven Framework November 2026 Release Notes

Fabric Metadata Driven Framework
Table of Contents
Microsoft Fabric Metadata Driven Framework November 2026 Release
One of the most common questions I get about the Fabric Metadata-Driven Framework (FMD) is: "How can I load some tables every hour and the rest only once a day?"
Schedule Groups in the FMD Framework: load the right entities at the right time
Until now, PL_FMD_LOAD_LANDINGZONE picked up every active landing-zone entity in the workspace. That works well when everything runs on the same cadence. In practice, though, sales orders may need to be refreshed hourly while reference data only changes overnight.
Another frequent request from the community was to load entities by source system or by department. When several teams or sources share one FMD deployment, you want to run each one independently, so the solution can scale out without one large load that does everything.
The new Schedule Groups feature solves both. You keep all metadata in the configuration database, group entities by schedule, source, or department, and choose which group to run when you start the pipeline. You can now create several schedules on the PL_FMD_LOAD_ALL pipeline, each with its own ScheduleGroupName parameter value.
What is a Schedule Group?
A schedule group is a named set of landing-zone entities, such as Hourly, Nightly, or Finance. When you start a pipeline, you pass the name of the group, and only the entities in that group are loaded.
A few design principles:
- An entity can belong to multiple groups. For example, a table can be in both Hourly and Finance. It appears only once in each group.
- The Default group always exists. Every landing-zone entity is a member, so existing deployments keep working without changes.
- The schedules themselves still live in Fabric. A schedule group decides what is loaded. A Fabric pipeline schedule decides when.
What changed?
1. A new ScheduleGroupName pipeline parameter
Every Fabric Data Pipeline in the framework, except PL_FMD_LOAD_BRONZE and PL_FMD_LOAD_SILVER, now has a string parameter named ScheduleGroupName that defaults to Default.
Each pipeline passes the value unchanged to the Fabric pipelines it invokes. For example, if you start PL_FMD_LOAD_ALL with ScheduleGroupName = Hourly, every landing-zone command and copy pipeline in the chain receives Hourly.
Bronze and Silver are not filtered by schedule group. They continue to process whatever has landed and been queued.
2. Two new tables in the configuration database
| Table | Purpose |
|---|---|
| integration.ScheduleGroup | One row per group: ScheduleGroupId and a unique ScheduleGroupName |
| integration.ScheduleGroupEntity | The relationship between a schedule group and a landing-zone entity. (ScheduleGroupId, LandingzoneEntityId) is the primary key. |
3. The landing-zone view now includes the group
execution.vw_LoadSourceToLandingzone now joins the schedule-group tables and exposes ScheduleGroupId and ScheduleGroupName. Every lookup that reads this view, 21 in total, now filters by the pipeline parameter:
... AND ScheduleGroupName = N'@{replace(pipeline().parameters.ScheduleGroupName, '''', '''''')}'
Apostrophes in group names are escaped, so a name such as Erwin's tables works correctly.
Backward compatibility: the Default group
I didn't want existing deployments to require any manual work. That is handled in three ways:
- On deployment, the DACPAC post-deployment script runs integration.sp_InitializeDefaultScheduleGroup. It creates the Default group if it doesn't exist and adds all existing landing-zone entities, including inactive ones. It is idempotent and never removes custom memberships.
- On every new insert, the trigger integration.tr_LandingzoneEntity_DefaultScheduleGroup adds the new entity to Default. This applies whether the entity is created through sp_UpsertLandingzoneEntity, sp_UpsertLandingzoneBronzeSilver, a direct INSERT, or a multi-row insert. The trigger runs in the same transaction, so rolling back the insert also rolls back the membership.
- Pipeline parameters default to Default, so schedules that don't pass the parameter continue to load the same active entities as before.
An unknown or empty group name returns no entities; it does not fall back to loading everything. If you have saved runs or schedules that explicitly pass an empty string, change the value to Default.
Managing groups: two new stored procedures
Create or rename a group
-- Create the group, or return the existing one
EXECUTE [integration].[sp_UpsertScheduleGroup] @ScheduleGroupName = N'Hourly';
-- Rename a group by its id
EXECUTE [integration].[sp_UpsertScheduleGroup]
@ScheduleGroupId = 2,
@ScheduleGroupName = N'EveryHour';
The procedure is idempotent and returns the group row. You cannot rename the Default group, because the insert trigger relies on that name.
Add an entity to a group
EXECUTE [integration].[sp_UpsertScheduleGroupEntity]
@LandingzoneEntityId = 42,
@ScheduleGroupName = N'Hourly';
The combination of landing-zone entity and schedule group is unique, so running the procedure again doesn't create a duplicate. It fails with a clear error if the entity or the group doesn't exist.
To add a whole data source at once:
DECLARE @Id BIGINT;
DECLARE c CURSOR LOCAL FAST_FORWARD FOR
SELECT LandingzoneEntityId
FROM integration.LandingzoneEntity
WHERE DataSourceId = 3;
OPEN c;
FETCH NEXT FROM c INTO @Id;
WHILE @@FETCH_STATUS = 0
BEGIN
EXECUTE integration.sp_UpsertScheduleGroupEntity
@LandingzoneEntityId = @Id,
@ScheduleGroupName = N'Hourly';
FETCH NEXT FROM c INTO @Id;
END;
CLOSE c;
DEALLOCATE c;
Putting it together: hourly and nightly loads
- Create the groups:
EXEC integration.sp_UpsertScheduleGroup @ScheduleGroupName = N'Hourly';
- Add the fast-changing entities to Hourly with sp_UpsertScheduleGroupEntity.
- In Fabric, add two schedules to PL_FMD_LOAD_ALL:
- Every hour, with ScheduleGroupName = Hourly
- Every night, with ScheduleGroupName = Default, which still contains everything
- Check which entities a run will pick up:
SELECT EntityId, SourceName, ScheduleGroupName FROM execution.vw_LoadSourceToLandingzone WHERE ScheduleGroupName = N'Hourly';
Because every entity also remains in Default, the nightly run acts as a full safety net, while the hourly run keeps your critical tables fresh.
Scaling out: one schedule per source or department
The same approach works for organizing loads by source system or department, instead of only by time. Create a group for each one, for example SAP, Salesforce, Finance, and HR, and add the relevant entities to it.
Then, in Fabric, open PL_FMD_LOAD_ALL → Schedule and add one schedule per group. In each schedule, set the ScheduleGroupName parameter to the group:
| Schedule | Recurrence | ScheduleGroupName |
|---|---|---|
| SAP load | Every hour | SAP |
| Salesforce load | Every 4 hours | Salesforce |
| Finance load | Daily at 06:00 | Finance |
| HR load | Daily at 02:00 | HR |
| Full safety net | Nightly | Default |
Each schedule runs the same pipeline chain but loads only its own entities. This brings several benefits:
- Independent runs. A failing source doesn't block the other sources or departments.
- Easy onboarding. Adding a new source or department means creating a group and a schedule, not a new pipeline.
- Better capacity planning. You can spread loads across the day instead of running one large batch.
- Clear ownership. Each department can see and manage its own group of entities.
Upgrading
Redeploy the framework database and pipelines together using NB_SETUP_FMD.ipynb. Deploy both at the same time: the new pipelines require the new view columns, and the old pipelines don't pass a schedule group. The DACPAC creates the tables, updates the view, adds the procedures and trigger, and initializes the Default group automatically. No manual SQL is required.
See the deployment guide for details.
Wrap-up
Schedule Groups let you run different load cadences from the same metadata-driven pipelines, without duplicating pipelines or adding custom filtering. Existing deployments keep working because of the Default group, and new groups take just two stored-procedure calls.
As always, feedback, issues, and pull requests are welcome on GitHub. Find the full documentation at erwindekreuk.com/fmd-framework.
New to the Microsoft Fabric Metadata Driven Framework?
Learn more about the framework architecture, deployment model, and features here:
The Microsoft Fabric Metadata Driven Framework continues to help organizations standardize data platform delivery, automate Microsoft Fabric deployments, improve governance, and accelerate AI-ready architectures. With every release, the framework becomes more reliable, secure, and easier to adopt for enterprise-scale Microsoft Fabric implementations.
GitHub Repository
FMD_FRAMEWORK
Hit Subscribe to get future release notes delivered straight to your inbox!


