Assign and Provision users and groups in the Enterprise Application
In the previous blogyou learned how to configure the Enterprise Application. In this blog, you will learn how to assign and Provision Users and Groups.
Once the Users and groups are assigned to the Enterprise application you can provision the Users and groups to your Azure Databricksaccount or Azure Databricks Workspace.
Add users and groups
Click on the Add user/group in the Enterprise application on the left pane to add the required users and groups.
When you see above message, that means that you don't have a Premium Azure Active Directory edition account. Don't worry, you can still provision users, for Groups you to need a Premium edition.
Note: If you have existing Azure Databricks workspaces, in case you sync on Account Level, make sure that you add all existing users and groups in those workspaces to the above Enterprise application.
Start the provisioning
The last step is to provision the users and the groups. The provision will automatically sync the assigned users and groups to your Azure Databricks account.
Go back to the provisioning option on the left pane.
Mappings
Enable the user and group sync option in the mappings section.
Settings
Set the scope to Sync only assigned users and groups, otherwise all your users in your Azure Active Directory will be synced, which is not necessary
The next step is, set the Provisioning Status toggle to on.
After a few minutes your users will be synced.
There are 2 more options which we can set:
Notification Email: Send an email notification when a failure occurs
Prevent accidental deletion: Set a threshold for Accidental deletion more on how this works can be found here.
Checking the Provisioning Logs
Once the provision of the users and groups has been done, you can check the details in the provision logs.
Click on the left side provisioning:
The details of the provisioning should be visible now, good to know that the interval of syncing is fixed to 40 minutes.
Click on the View provisioning logs to see a detailed overview of the sync.
Tips and tricks for Provisioning
The interval of syncing is fixed and set to 40 minutes, the initial one is directly started.
The username or email address of an Azure Databricks workspace user cannot be updated.
The admin group cannot be used as Group name.
Groups cannot be renamed in Azure Databricks or in the Azure Active Directory.
Nested groups or service principals cannot be synced.
Blog Serie: Provisioning identities from Azure Active Directory to Azure Databricks.
Instead of adding users and groups manual to your Azure Databricks environment, you can also sync them automatically from your Azure Active Directory to your Azure Databricks account with SCIM. This is one of the recommendations from Databricks.
Other advantages are:
Stream less onboarding of new employees or teams in Azure Databricks.
Users can be easily deleted from the Azure Databricks workspaces through the Azure Active Directory. This ensures a consistent offboarding process and prevents unauthorized users from accessing sensitive data.
Their are a couple of important requirements to have in place before we can start, you need to have or be:
Azure Databricks account with a Premium Plan.
Azure Databricks account admin to provision users to your Azure Databricks account using SCIM.
Azure Databricks workspace admin to provision users to an Azure Databricks workspace using SCIM.
Azure Active Directory account must be a Premium edition account to be able to provision groups.
Provisioning of users is available for all Azure Active Directory editions (including the Azure AD Free)
Blog Serie
This blog post series contains the following topics, which I will post in the next few days:
There are 2 different options to provision users and groups to Azure Databricks using Azure Active Directory (AAD) at the Azure Databricks account level or at the Azure Databricks workspace level. This post is related to the Azure Databricks Account Level.
Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
Make sure that you're an Azure Databricks account admin. If you're not an account admin, check who is an account admin( you see this on the main page of the user Management option). Ask the Account admin to grant you access, they can do this by clicking on the account name.
Once you're Account Admin, click on the left side, click on the user setting icon(red).
Click on User Provisioning and click on set-up user provisioning.
Copy the SCIM token and the Account SCIM URL and store them in an Azure Key Vault. We need these settings later to configure the Enterprise Application.
Configure the Enterprise Application
In the Azure portal, go to Azure Active Directory > Enterprise Applications.
Click on new application and search for the "Azure Databricks SCIM Provisioning Connector"
Click on the app:
Enter a Name for the application, I used Azure Databricks SCIM AzureDataBricksWestEurope
Click on Create and wait until the application is created.
Click on Provisioning and set Provisioning Mode to Automatic.
Set the Tenant URL to the Account SCIM URL that we saved earlier in our Key Vault.
Set Secret Token to the Azure Databricks SCIM token that we generated and saved earlier in our Key Vault.
Click on Test Connection so see if everything is configured correctly.
In my next blogI will explain how to Assign and Provision users and groups in the Enterprise Application(SCIM).
Connect and Manage Azure Databricks in Microsoft Purview
This week the Purview team released a new feature, you’re now able to Connect and manage Azure Databricks in Microsoft Purview.
This new functionality is almost the same as the Hive Metastore connector which you could use earlier to scan an Azure Databricks Workspace. This new connector is an easier way to setup scanning for your Azure Databricks Workspace.
Tables including the columns, foreign keys, unique constraints, and storage description.
Views including the columns and storage description.
Fetching relationship between external tables and Azure Data Lake Storage Gen2/Azure Blob assets.
Fetching static lineage on assets relationships among tables and views.
Let’s have a look how to setup this connector, before you can start make sure you have the following Prerequisites in place:
Microsoft Purview account with Data Source Administrator and Data Reader permissions.
Self-Hosted Integration Runtime.
Personal access token in Azure Data Bricks.
Cluster in Azure Data Bricks.
Register the Azure Databricks Workspace
Select Data Map on the left pane and select Sources.
Select Register.
In Register sources, select Azure Databricks and click on Continue.
On the Register sources (Azure Databricks) screen, do the following:
Enter a name that Microsoft Purview will list as the data source.
Select the subscription and workspace that you want to scan from the dropdown list.
Select a collection.
Setup the Integration Runtime
Select Data Map on the left pane and select Integration Runtime.
Click on New.
Select the Self-Hosted.
Enter a name and description, click on create.
Copy the authentication key.
Configure the Self-Hosted Integration Runtime
On an Virtual Machine in Azure:
JDK 11, must be installed, not another version, I tried it with 17 and 19 this is not working. To download the JDK 11, you need to have an Oracle account.
After installing the SHIR paste the authentication key in the Gateway to start the communication. Wait before the communication start.
Reboot your Virtual Machine to continue.
After rebooting, Select Data Map on the left pane and select Integration Runtime and check if the SHIR is running.
Setup the Scan
The last step to configure is the scan.
Select Data Map on the left pane and select Sources and select the Azure Databricks you just created.
Select New Scan.
Name, create a logical name for your scan. Weekly, Monthly, Once or a different name. TIP, add your clustername or id to the scanname. You need to create a scan for every cluster in an Azure Databricks workspace. This way you can see the difference between the clusters.
Connect via IR, select the SHIR you just created.
Credential, select the Personal Acces token, which is stored in de Azure KeyVault.
Cluster ID, Specify the cluster ID that Microsoft Purview need to connect to, to perform the scan.
Mount Point, if you have external storage manually mounted to Databricks, you provide the locations here. Use the following format /mnt/<path>=abfss://<container>@<adls_gen2_storage_account>.dfs.core.windows.net/.
Maximum memory available: Specify the maximum memory available in GB to be used by scanning processes. If the field is left blank, 1 GB will be considered as a default value.
The default location of the cache in your VM is C:WindowsServiceProfilesDIAHostServiceAppDataLocalMicrosoftAzureDataCatalogCache. Unselect the checkbox if you want cache to be stored in a different location.
Click on continue.
Select the trigger you want. Click on save and run.
Check if the scan starts, be aware that the scan will trigger your Azure Databricks cluster to start.
Browse and search assets
Once the data is scanned you can browse and search the Metadata.
Select Data Catalog on the left pane and select Browse Assets.
From the Databricks workspace asset, you can find the associated Hive Metastore.
Select the Azure Databricks and click on edit details on the right side.
Click on Hive Metastore, on the Related tab you can see the Hive DB and the assets. Click on one of the assets to see the lineage when applicable.
Conclusion
The first steps towards a Native integration of Azure Databricks is now available in Microsoft Purview, but we're not there yet. If you want to have a more extensive lineage and can read more details from the Notebooks execution including Delta Lake than, I advise you to use the Azure Databricks to Purview Lineage Connector.
In the notes of this Solution Accelerators, is noted "With native models in Microsoft Purview for Azure Databricks, customers will get enriched experiences in lineage such as detailed transformations." So hopefully we can expect more in the future.
Be aware that lineage is available at the asset level not at column level, hopefully that will arrive soon.
In the notes of the above Solution Accelerators, is noted "With native models in Microsoft Purview for Azure Databricks, customers will get enriched experiences in lineage such as detailed transformations." So hopefully we can expect more in the future.
Like always in case you have questions, do not hesitate to contact me.
It's that time of year again to reflect on the past year. Also think it's really good, to see what you've done in the past year. It is also the time again to traditionally bake Oliebollen on this day, a Dutch Tradition that we do on New Year's Eve.
This year we celebrate New Year's Eve with the family, we were supposed to go to relatives, but unfortunately these have all been felled by the flu. We will enjoy an evening of games, oliebollen and some drinks.
Looking back at 2022, I can say that I personally had a great year. Finally we are released from all COVID restrictions, we can go to physical events again, back to our customers, back to the office and see many people in real life again. It was a very busy year in terms of work, this is also one of the main reasons that I have blogged much less than other years. The inspiration and energy is slowly coming back for this, so with some hope I can change this soon.
And as many would say, I should exercise more in the evenings and not always sit behind my laptop. My work is my hobby, so that will be difficult, but it will be one of my New Year's Good intentions for 2023. A better work life balance would be better for my health and maybe I should seek help in the form of a coach for that. If you have suggestions for me, let me know.
Speaking/Volunteering
This year I have spoken at several events on the topics of Azure Synapse Analytics and Microsoft Purview.
DataMinds (Virtual)
Did a virtual session for the DataMinds UG on Data Governance with Azure Purview( Yes is was on that moment still Azure Purview and not Microsoft Purview)
Data Minutes (Virtual)
The second event of the year, within my team I proposed to attend this event together. During the event there was a Last-Minute cancellation, at the request of Ben Weissman I gave a short session about Access Control in Azure Synapse. It's a fun event where you have a bunch of 10 minute blocks. Works very inspiring.
Data Toboggan (virtual)
Talked about access control in Azure Synapse Analytics. Data Toboggan is one of the events that is 100% focused on Azure Synapse Analytics.
SQL BITS
This year I volunteered at SQLBits for the first time and became part of the Orange family and as you can see in the picture it is a very big family. In addition to volunteering, I was also asked by Microsoft to present a session during SQL Bits on Cloud Scale Analytics solutions. Thanks again Tony and Wee Hyong for the invite.
SQLBITS Orange Team 2022
Data Saturday Stockholm
My first time in Stockholm, great event. Bra så alla tillbaka personligen This means great, so everyone back personally.
DataGrillen
When we say: Data, bratwurst and beer, we are of course talking about DataGrillen. After more than 2 years of absence, it was that time again in recent days, with speakers from all over the world, beautiful weather and a large group of participants. And as usual with this event, the first day ends with a barbecue for all participants. By the way, I can't imagine a better event to celebrate my 50th birthday. It is therefore a great pity that there will be no DataGrillen next year. Hopefully again soon, in any form.
Datagrillen 2022
Scottish Summit
First time in Glasgow, second time speaking at the Scottish Summit on one of my favorite topics Data Governance and Purview.
DATA Scotland
Second time in 3 months coming to Glasgow and this time it was sunny, over 400 people in attendance and over 50 sessions.
DATA SCOTLAND 2022
DataMindsConnect
This year I volunteered for 2 days, the event was sold out with over 650 attendees. For us as Dutch people, the event is easily accessible by car, which means that many Dutch people are also present as participants.
Experts Living in the Netherlands
This was a very big IT event in the Netherlands, my employer InSpark was one of the sponsors. But even better to mention that during this event I was allowed to give a session with my colleague Albert, this was the first time that I presented together with a colleague. And this is certainly worth repeating.
Pass Data Community Summit
I've always wanted to speak at the Pass Data Community Summit and it's a dream come true and definitely one of my bucket list items. I had 3 sessions on Microsoft Purview. The event was great again, had great conversations and met so many old and new people again. Nice thing about this event was that a MVP pre day was also organized on the Microsoft Campus, thanks again to Rie Merritt for the entire organization and putting together the agenda. It always remains a wonderful feeling to be on the Campus. Hopefully we can go to Seattle again next year. And let's not forget the day that as a community we showed our support to Hugo Kornelis #teamhugo. This was a very and beautifully moving moment, all messages on the socials, all blue shirts. Hugo is fighting Acute Myeloid Leukemia (AML.)
Pass Community Summit 2022
The above was not possible with the support from my wife and kids, you are also quite often on the road during the weekends and I would like to thank my employer InSpark very much as well, they are the ones who encourage and financially support all our MVPs, but also other colleagues, to speak at international events.
InSpark
At my employer InSpark, we have made great progress with the Data and AI team this year with Managed Oyxgen, our Data Platform as a Service. The 2023 roadmap for our solution looks impressive and I'm already looking forward to working on it with everyone. 👉 Within our team we can be proud that we: ✅ Having challenging and varied work. ✅ Added even more gears to our accelerators. ✅ Creating lots of space for your own input, innovation and creativity. ✅ Working and contributing to many interesting projects. ✅ Having a good working atmosphere. ✅ Even took yoga classes. ✅ Attend the coolest events and even speak there.
I am therefore very proud of the team that we have all achieved together this year, on to even more beautiful and innovative projects
And the above certainly contributes to the fact that I can close 2022 as a great year and I look forward to 2023 with great pleasure and enthusiasm.
I wish everyone a great 2023, on to a successful year
All sessions of SQLBits 2022 have been made available to everyone and can now be viewed via their Youtube channel. Microsoft asked me to present me this session during SQL Bits in the Cloud Scale Analytics solution area.
Session Title:
Lake Database with Database Template and Mapping Data with Azure Synapse Analytics
Description:
Database templates in Azure Synapse Analytics are blueprints which can be used by organizations to plan, architect and design solutions.
How can we use these Database Templates in a day-to-day business, in order to speed up to automate this process? Map data tool can help us with that. The map data tool can generate a mapping data flow without having to start from a blank canvas. In this presentation, you will see how this all works in a step-by-step demo-based session.
During SQL Bits the Mapping Data tool was still in Preview, the great news is that this functionality is now GA.
SAVE THE DATE
SQLBits 2023 will back next year 14 - 18 March 2023, so mark you calendars.
SQLBITS Orange Team 2022
Datagrillen 2022
DATA SCOTLAND 2022
Pass Community Summit 2022
Rotterdam Vuurwerk Erasmusbrug.