Get ready for the next enhancement in Microsoft Purview
Get ready for the next enhancement in Microsoft Purview, as it brings a range of exciting new features and capabilities. To ensure the best experience with Purview, it is recommended that you tag your existing Microsoft Purview accounts appropriately.
Introducing modern data governance for the era of AI
Check the schedule below when the new Experience will reach your region. Make sure you have enabled the new portal experience if not check out my blog postMicrosoft Purview new Experience is coming
How Microsoft Purview and Microsoft Fabric work together to empower data discovery and governance
Microsoft Purview is a unified data governance service that helps you manage and govern your on-premises, multi-cloud, and software as a service (SaaS) data. Microsoft Fabric is a new cloud-based data platform that enables you to create, share, and collaborate on data-driven insights with your team. Together, Microsoft Purview and Microsoft Fabric offer a seamless integration that allows you to browse and search Fabric items, access metadata from Fabric items, and apply data policies and classifications to Fabric items.
New Portal Experience
A few months ago, Microsoft announced the new portal Experience in Microsoft Purview as it offers a range of exciting new features and capabilities. Data Governance, Risk and Compliance are increasingly integrating into a unified experience. Microsoft Fabric will have a native integration with Microsoft Purview.
Browse and search Fabric items
Just like Microsoft Azure, Microsoft Fabric is a new source for Microsoft Purview. Since Microsoft Purview is attached to every Fabric instance by default, you can click on the tile “Microsoft Fabric” on the front page of Microsoft Purview Data Catalog to start browsing your Fabric items. Automatically, any user can see the workspaces and Fabric items based on the permission setting they have in Fabric. You can also use the search bar to find Fabric items by keywords, filters, or facets.
Access metadata from Fabric items
In the coming weeks, Microsoft Purview Enterprise customers can provide broader access to metadata from Fabric items by scanning Fabric. When a Fabric is scanned, Microsoft Purview writes information about Fabric items to the Purview data map, and access to that metadata is governed by Microsoft Purview access control. This allows administrators to give users metadata access for data discovery or governance, without requiring those users to have read permissions on the underlying data sources.
Live view in Microsoft Purview
Resources in live view in the Microsoft Purview Data Catalog automatically have this metadata available:
Name
Properties
Schema
Lineage
Creating a new workspace in Fabric will automatically appear in Microsoft Purview.
Available resources
The following Fabric items will be available in Microsoft Purview as part of this public preview release.
Experiences
Fabric items
Real-Time Analytics
KQL Database
KQL Queryset
Data Science
Experiment
ML Model
Data Factory
Data pipeline
Dataflow Gen2
Data Engineering
Lakehouse
Notebook
Spark Job Definition
SQL analytics endpoint
Data Warehouse
Warehouse
Power BI
Dashboard
Dataflow
Datamart
Dataset
Report
Paginated report*
* Only available by scanning
Conclusion
Microsoft Purview and Microsoft Fabric are two powerful services that work together to empower data discovery and governance. By integrating Microsoft Purview and Microsoft Fabric, you can leverage the benefits of both services, such as:
Browse and search Fabric items in the Microsoft Purview Data Catalog
Access metadata from Fabric items without requiring data access permissions
If you want to learn more about Microsoft Purview and Microsoft Fabric, you can visit the following links:
Get ready for the next enhancement in Microsoft Purview
Get ready for the next enhancement in Microsoft Purview, as it brings a range of exciting new features and capabilities. To ensure the best experience with Purview, it is recommended that you tag your existing Microsoft Purview accounts appropriately.
Mark your Purview accounts with the proper tags
In short, mark each Purview account with the following tags: Name: "Purview Environment," Value: "Production."
In the long story, you have several tag options available to you. The table below outlines the different tags and their purposes:
In all purposes the tag name is "Purview Environment"
Production - This tag signifies that the account is or will be used for cataloging and governance requirements in a production environment. It is a candidate to be selected as the primary account for the tenant.
Pre-production - This tag indicates that the account is or will be used to validate cataloging and governance requirements before making them available in production. It is a candidate to be merged as a domain or can be deleted.
Test - This tag suggests that the account is or will be used for testing capabilities in Microsoft Purview Governance. It is a candidate to be merged as a domain or can be deleted.
Dev - This tag signifies that the account is or will be used to test capabilities or develop custom code and scripts in Microsoft Purview Governance. It is a candidate to be merged as a domain or can be deleted.
Proof Of Concept - This tag indicates that the account is or will be used to test capabilities or develop custom code and scripts in Microsoft Purview Governance. It can be deleted in the future.
Deprecate - This tag is for accounts that were created a while back but are not in use today. They can be deleted in the future.
Create tag
The following instructions will allow you to successfully tag your Purview account(s) with the desired classification:
1. Sign in to the Azure portal using your Azure account credentials.
2. Use the search bar to find and select your Microsoft Purview account.
3. Locate the Tags (edit) section on the left of the Purview account overview page.
4. Click on the Tags (edit) link to open the tags editor.
5. Add a new tag with a Name and Value according to the provided options (e.g., Name: "Purview Environment," Value: "Production").
6. Click on Save to apply the tag to your Purview account.
What's Next!
The next Microsoft Purview enhancement is coming to you in a few weeks. Along with the features you've enjoyed with Microsoft Purview so far, this enhancement will provide these additional capabilities:
Centralized organization-wide data governance that automatically gives you visibility across your Microsoft Cloud.
Configuration and set up is no longer required to capture metadata for Microsoft Cloud - Microsoft Purview is auto-attached to Microsoft Fabric and Azure SQL.
A clean, crisp, and more intuitive user interface to navigate the platform and apps.
These new features will be turned on automatically and added to your existing capabilities.
How can you use our new experience?
These new features will be turned on automatically and seamlessly integrated with your existing capabilities. The new experience will be available once your organization has been enabled. The exact steps to get started will depend on your organization's current structure, and more information will be provided in the coming weeks.
Introduction to Real-Time Analytics in Microsoft Fabric
Real-Time Analytics is one of the data and analytical workloads/experiences available in Microsoft Fabric, the new platform currently in Public Preview at Microsoft. With Real-Time Analytics, companies and developers can gain valuable insights and analysis from real-time data streams.
A unified analytics solution for the era of AI
Microsoft Fabric brings a unified SaaS-based solution that stores all organizational data where analytics workloads operate. Microsoft Fabric brings together existing offerings such as Data Factory, Azure Synapse Analytics, and Power BI into one unified product for all data and analytics workloads.
Key pilars:
Complete analytics platform
Lake centric and open
Empower every Office user
AI Powered
When Microsoft Fabric is not yet activated in your tenant, you can activate it in the Admin Portal. Please note that Microsoft Fabric Capacity(Trial) or Power BI Premium Capacity is required to get started with Microsoft Fabric.
Microsoft Build
Now that we have seen the initial sessions during Microsoft Build, it's time to delve deeper into a topic. But what an announcement! We have all worked hard on this in the last couple of months. We have done a lot of testing and provided a lot of feedback. And personally, I can say that all feedback has been listened to carefully.
In this blog, I will delve deeper into Real-Time Analytics, one of the available experience in Microsoft Fabric. An experience is a look and feel of various Fabric Artifacts for a specific role such as a Data Engineer, Data Analyst or Data Scientist. For all available experiences see picture above.
Real-Time Analytics
Real-Time Analytics is critical in today's fast-paced business environment. It enables organizations to react immediately to events and trends as they happen, rather than reacting to historical data afterwards. The Real-Time Analytics workload allows users to monitor, analyze, and visualize data in real-time to make fast and data-driven decisions.
Here are some key features and functionalities of Real-Time Analytics in Microsoft Fabric:
Real-time data processing: The workload supports processing large amounts of data in real-time, giving users instant access to up-to-date information.
Advanced analytics: Built-in analytics capabilities enable users to apply complex calculations and statistical models to real-time data for deep insights.
Flexible visualizations: The app offers a wide range of visualization options, such as graphs, charts, and dashboards, to present data in a clear and understandable manner.
With Data Activator(coming soon): Users can set up custom notifications and alerts based on predefined criteria, keeping them informed of important events or anomalies in real-time.
As you can see, you can use Real-Time Analytics for a range of solutions, such as IoT analytics, Telemetry data, human and system logs and in many scenarios including manufacturing operations, cybersecurity, oil and gas, automotive and many more.
Benefits
One of the great benefits of using Real -ime Analytics in Microsoft Fabric is that you have a seamless integration with other artifacts in Fabric such as Lakehouse, Data Warehouse and Machine Learning Models for Predictive Analytics. One of the other benefits in Microsoft Fabric is that you don’t have to start from scratch, is very easy to connect to existing Event Hubs to load your streaming events into Fabric. Which I will explain in my next blog.
Real-Time Analytics Artifacts
Currently the Real-Time Analytics workload supports 3 different artifacts:
KQL Database: A Kusto database exactly the same as you were used to in Azure Data Explorer
KQL Queryset: Collection of queries which you can run on top of your KQL Database
Eventstream: Capture, transform and route real-time event stream to various destinations with a no-code experience. Similar to Azure Stream Analytics
OneLake: The foundation for Microsoft Fabric
OneLake eliminates today’s pervasive and chaotic data silos by providing a data lake as a service without you needing to build it yourself. OneLake is the OneDrive for data and like OneDrive, OneLake is provisioned automatically with every Fabric tenant with no infrastructure to manage. All Fabric Artifacts, such as mentioned above for Real-Time Analytics are deployed/ provisioned automatically into the Onelake upon on creation. How easy is that?
Having a closer look at the picture above, you see an end to end workflow for a Real-Time Analytics scenario.
Ingest the data from Event Hub, custom apps, structured and Unstructured data source with pipelines and Dataflows.
Store the data in a KQL Database or Lakehouse.
Expose the data in Power BI and/or make available in Notebooks and KQL Queriesets.
Train and test the data with Machine Learning Models and Experiments.
With this end to end workflow you can directly see which artifacts you need to use to build your Real-Time Analytics Solution.
Public Preview
It's important to note that as Microsoft Fabric is currently in Public Preview, additional functionality is still being developed, and feedback is being incorporated. This presents a great opportunity for users to get involved early, provide feedback, and contribute to the further development of Microsoft Fabric.
When you decide to start using Microsoft Fabric and encounter any issues with the Real-Time Analytics workload, please don't hesitate to reach out to me. I’m here to assist and appreciate your feedback to further enhance the platform.
Click below to read more about Microsoft Fabric and Real-Time Analytics.
In my next blog I will get a bit deeper how easily you can connect existing Event Hubs to Microsoft Fabric. So stay tuned(published on may 26th 2023)
Note:
Please be aware that Microsoft Fabric is currently not authorized for production use as it is still in the Public Preview phase. It's important to consider this when planning deployments or making critical business decisions.
In the video below, Tzvia Gitlin Troyna, a Principal Manager with Synapse Real-Time Analytics experience in Microsoft Fabric, shares a first look at what's included in the first release of Real-Time Analytics in Microsoft Fabric.
Microsoft Purview Pricing and introduction of Purview Applications
The Microsoft Purview pricing page has been updated. Below I have listed most of the changes. The most important changes are the introduction of the Microsoft Purview Applications and the pricing of the Insights Generation. The standard level of 1 capacity unit of 2 GB metadata storage and 25 operations per sec has been increased to 10 GB.
Post has been updated on April 25th.
Microsoft Purview Data Map
The Microsoft Purview Data Map stores metadata, annotations and relationships associated with data assets in a searchable knowledge graph.
Data Map is billed across three types of activities:
Data Map Population– examples include metadata & lineage extraction or classification based on metadata & content inspection.
Data Map Enrichment– examples include use of resource sets to optimize storage of data lake assets, or aggregation of classifications to generate insights
Data Map Consumption- examples include serving up search results or rendering lineage graph. This also includes the use of Apache Atlas API to build apps on Data Map.
Data Map Population
Automated Scanning, Ingestion & Classification
Data Map population is serverless and billed based on the duration of scans (includes metadata extraction and classification) and ingestion jobs. Automated scans using native connectors trigger both scan and ingestion jobs. Push based updates from a Microsoft Purview client (e.g., lineage push from Azure Data Factory or Azure Synapse Analytics) only trigger ingestion jobs.
Price
For Power BI online
Free for a limited time
For SQL Server on-prem
Free for a limited time
For other data sources
€0.582 per 1 vCore Hour
Data Map Enrichment
Advanced Resource Set
Advanced Resource Set is a built-in feature of the Data Map used to optimize the storage and search of data assets associated with partitioned files in data lakes. Billing for processing the resource set data assets is serverless and based on the duration of the processing, which can vary based on the change in partitioned files and resource set profile configured. In the Management Center you have an option to toggle on or off.
Note: By default, the advanced resource set processing is run every 12 hours for all the systems configured for scanning with resource set toggle enabled.
Price
Advanced Resource Set
€0.194 per 1 vCore Hour
Insights Generation
Insights Generation aggregates metadata and classifications in the raw Data map into enriched, executive-ready reports that can be visualized in the Data Estate Insights application and granular asset level information in business-friendly format that can be exported. Report visualization and export incurs charges from Insights Report Consumption in the Data Estate Insights application.
Price
Report Generation
€0.758 per 1 vCore Hour
Insight Generation is new for me, currently it looks like around €70,00.
Note: By default, Insights Generation is enabled and provisioning and can be turned off in the Management center of Microsoft Purview governance portal. In the Management Center you have now an option to toggle on or off the Insight Generation. If the toggle is on and the report frequency is off than you can still see the reports with the latest report generation. If set to automatic your reports will refreshed based on your scanning and activities in de Portal. Currently the automatic refresh is weekly.
If the toggle is off the Insight Generation activity will you give you the following warning:
Data Map Consumption
Elastic Data Map
By default, a Microsoft Purview account is provisioned with a Data Map of at least 1 Capacity Unit. 1 Capacity Unit supports requests of up to 25 data map operations per second and includes storage of up to 10 GB of metadata about data assets.
Price
Capacity Unit
€0.380 per 1 vCore Hour
Note: The storage size was until last week 2 GB for 1 capacity Unit and has been resized to 10 GB. so that is a major change.
Microsoft Purview Applications
Microsoft Purview Applications are replacing the C0, C1 and D0 options which we had previously. Microsoft Purview Applications are a set of independently adoptable, but highly integrated user experiences built on the Data Map including Data Catalog, Data Estate Insights and more. These applications are used by data consumers, producers, data stewards and officers that enable enterprises to ensure that data is easily discoverable, understood, high quality, and all use is per corporate and regulatory requirements.
Data Catalog
Data Catalog is an application built on Data Map for use by business users, data engineers and stewards to discover data, identify lineage relationships and assign business context quickly and easily.
Price
Search and browse of data assets
Included with the Data Map
Business Glossaries
Included with the Data Map
Lineage Visualization
Included with the Data Map
Self-Service Data Access
Free in preview
Data Estate Insights
Price
Insights Consumption
€0.194 per API call
Note: Insights consumption is billed per API call. One API call returns up to 10,000 rows of tabular result. Like Insight Generation I've no idea yet what this will do with the cost. As soon this is available I will update this article.
Data Access Policies for SQL and Data Lakes(preview)
Data owners can centrally manage thousands of SQL Servers and data lakes to enable quick and easy access to data assets mapped in the Data Map for performance monitors, auditors, and data users.
Price
SQL DevOps access
Free in preview
Data Lake data asset access
Free in preview
Workflows(Preview)
Data owners and stewards can automate commonly used repetitive tasks associated with business processes like glossary curation and approval tracking using workflow management.
Price
Business Workflows
Free in preview
Data Sharing(Preview)
In-place Data Sharing lets users share data easily from within Microsoft Purview governance portal both within and between organizations, providing near real-time access to data without duplication.
Price
In place sharing for Azure Blob Storage and Azure Data Lake Storage (ADLS Gen2) storage accounts
Free
More details on data sharing in Microsoft Purview can be found here.
Pricing Example
Based on the example which is published on the pricing page, I've done a Calculation:
Example Scenario:
Data Map can scale capacity elastically based on the request load. Request load is measured in terms of data map operations per second. As a cost control measure, a Data Map is configured by default to elastically scale up to a peak of 8 times the steady state capacity.
For dev/trial usage:
Data Map (Always on): average of 2 capacity unit x Price per capacity unit per hour x 730 hours per month
Scanning (Pay as you go): Total duration (in minutes) of all scans in a month / 60 min per hour x 32 vCore per scan x €0.582 per vCore per hour
Resource Set: Total duration (in hours) of processing resource set data assets in a month * Price per vCore per hour
The total cost per month for Azure Purview = cost of Data Map + cost of Scanning + cost of Resource Set
Assuming above Scenario that we only use 1 Capacity Unit and use not more then 10 GB of Metadata storage and we scan our data once a week for 2 hours.
Data Map 2 CU x €0.380 X 730 hours = €554
Scanning 4 scans x 4 hours x 32 VCore x €0.582 per vCore per hour = €297
Resource Set 30 days x every 12 hrs x 8 Vcore x €0.194 per vCore per hour €93
In Total €944 including 4 scans, Data Estate Insight excluded. If you leave Microsoft Purview as is and no scanning you base fee will be €277 for 1 CU and Resource Set toggle need to be switch off
Data Estate Insights every week(4) x 8 Vcore x 4 hours x €0.758 = €97
Like always, in case you have questions, leave them in the comments or send me a message.