This Saturday I've been speaking during DataSaturday #4 Oslo. If you want to visit more Datasaturday events please visit the Data Saturdays event page.
Azure Purview
I presented a session on Azure Purview Microsoft's answer to Data Governance and Data Lineage
More clarity about pricing and when Azure Purview goes to GA is likely to become clear during the event on September 28. You can register for this event via the link below.
EVENT=>Achieve unified data governance with Azure Purview
As always, in case you have any questions, please feel free to contact me.
In case you have any questions left please feel free to ask them via the comment or Socials
This week the Azure Purview Product team added some new functionalities, new connectors(these connectors where added during my holiday), Azure Synapse Data Lineage, a better Power BI integration and the introduction of Elastics Data Map. Slowly we are on our way to a GA status, on September 2021, 28th there will be a Digital Event. Please find below some of announcements in detail.
New connectors in Azure Purview
Over the past period, the Azure Purview team has worked hard, they have already added the necessary new connectors such as ERWIN, Looker, Cassandra and Google Big Query.
This week it was time for some new functionalities.
Azure Synapse Analytics Data Lineage:
This functionality currently only works for a copy activity, but the first step has been made. Where for Lineage from Azure Data Factory you still had to make a link in Azure Purview, for the Lineage from Azure Synapse, it is the other way around. You create the link to Azure Purview in Azure Synapse. How to create this link I described this a couple of months ago in one of my post and can be found here.
Some known limitations on copy activity lineage based on the docs.
Currently, if you use the following copy activity features, the lineage is not yet supported:
Copy data into Azure Data Lake Storage Gen1 using Binary format.
Copy data into Azure Synapse Analytics using PolyBase or COPY statement.
Compression setting for Binary, delimited text, Excel, JSON, and XML files.
Source partition options for Azure SQL Database, Azure SQL Managed Instance, Azure Synapse Analytics, SQL Server, and SAP Table.
Source partition discovery option for file-based stores.
Copy data to file-based sink with setting of max rows per file.
Add additional columns during copy.
In additional to lineage, the data asset schema (shown in Asset -> Schema tab) is reported for the following connectors:
CSV and Parquet files on Azure Blob, Azure File Storage, ADLS Gen1, ADLS Gen2, and Amazon S3
Power BI supports now automated discovery of columns, measures and datatypes of the Power BI.
To enable this functionality you much enable the following settings in the Power BI tenant setting page(be aware that you need to be a Power BI Admin)
Allow service principals to use read-only Power BI admin APIs.
To use this setting create a Security group or use an existing one and add your Purview account to this SG.
Enhance admin APIs responses with detailed metadata
Elastic data map in Azure Purview
All Purview account created after August 2021, 18th are now created with the new Elastic data map concept. With this new concept your Purview account will come by default with one capacity unit and elastically grow based on usage. Each Data Map capacity unit includes a throughput of 25 operations/sec and 2 GB of metadata storage limit. So now when you’re not using Purview you’re not paying the default value of 4 capacity units.
The Data Map is billed on an hourly basis. You are billed for the maximum Data Map capacity unit needed within the hour. At times, you may need more operations/second within the hour, and this will increase the number of capacity units needed within that hour. At other times, your operations/second usage may be low, but you may still need a large volume of metadata storage. The metadata storage is what determines how many capacity units you need within the hour. Please read the documentation for a more detailed explanation and some examples
All existing Azure Purview accounts will be migrated in September/October to the Elastics data map concept.
The big question that remains open is what exactly does this Capacity Unit cost? For the time being during the Preview, it is still free, which can be read from the updated price page of Azure Purview..
More clarity about pricing and when Azure Purview goes to GA is likely to become clear during the event on September 28. You can register for this event via the link below.
EVENT=>Achieve unified data governance with Azure Purview
As always, in case you have any questions, please feel free to contact me.
This week the Azure Purview announced that they will extend the Azure Purview offer to provision 4 Capacity Units of the Data Map for free till May 31, 2021! Charging will start on June 1, 2021. Great news for customer who start exploring Azure Purview
More detail on Azure Purview pricing can be found here
New Sources
As you see above a lot of new Sources to scan have been announced and there will be more sources coming in the next months.
AWS Simple Storage Service (S3) data
SAP S/4 HANA
SAP ECC
Teradata
Oracle
And last but least, Azure Multiple. With Azure Multiple you can now govern multiple Azure sources within Azure Purview by registering an entire Azure subscription or a resource group.
And you setup scan and triggers to the discover assets within your subscription.
Also for Azure Multiple more resource types are coming in the upcoming weeks
Private Endpoint
You can now set up scans on your on-premises and Azure data sources with ingestion private endpoints and a self—hosted integration runtime to ensure network isolation, and metadata traversing only via private networks end-to-end and block all access to your Purview instance from public internet.
How do you integrate Azure Purview in Azure Synapse Analytics?
This article explains how to integrate Azure Purview into your Azure Synapse workspace for data discovery and exploration. Follow the steps below to connect your Azure Purview account in your Azure Synapse Workspace.
In the Management Hub you will see now a new option called Azure Purview.
Click on the option “Connect to a Purview Account”. Please be aware that you need a Contributor role in your Azure Synapse workspace and access to your Azure Purview Account(Purview Reader or Purview Curator).
Find the Purview account you want to connect to from the drop down list or add it manually by adding the source ID.
If the connection is successful, you will see the following screen. If not, make sure you have the correct role to connect to your Azure Purview account.
Data discovery:
If you select your Data, Develop or Integrate HUB, you will see in the top center a Search bar.
Using Azure Purview in Azure Synapse
To use Azure Purview in Azure Synapse it requires you to have access to the connected Azure Purview account. Azure Synapse will then passes-through the correct Azure Purview permissions.
Purview Reader role
Can read all content in Azure Purview
Purview Curator role
Can read all content in Azure Purview
Can edit assets, classification and glossary terms
Can apply classifications and glossary terms to assets.
Azure Purview actions
The following Azure Purview features are available in Azure Synapse Analytics(based on your role):
Overview of the metadata
View and edit schema of the metadata with classifications, glossary terms, data types, and descriptions
View lineage to understand dependencies and do impact analysis. For more information about, see lineage
View and edit Contacts to know who is an owner or expert over a dataset
Related to understand the hierarchical dependencies of a specific dataset. This experience is helpful to browse through data hierarchy.
Connect data to Azure Synapse
Add addition to above features, you can also connect directly to the assets you have searched.
Linked Service
Creating a new Linked Service will be required to copy data to Synapse or have them in your data hub (for supported data sources like ADLS Gen 2)
Integration Dataset
For objects like files, folders, or tables, you can directly create a new Integration Dataset and leverage an existing linked service if already created.
Develop in Azure Synapse
There are three actions that you can perform: New SQL Script, New Notebook, and New Data Flow.
SQL Script
View the top 100 rows in order to understand the shape of the data.
Create an external table from Synapse SQL database.
Load the data into a Synapse SQL database.
Notebook
Load data into a Spark DataFrame.
Create a Spark Table (if you do that over Parquet format, it also creates a serverless SQL pool table).
Data flow
Create an integration dataset that can be used as a source in a data flow pipeline.
These new functionalities makes the integration between Azure Purview and Azure Synapse Analytics even more Powerful. More details can be found here.
