Provision users and groups from AAD to Azure Databricks (part 1)

by Jan 17, 2023

Blog Serie: Provisioning identities from Azure Active Directory to Azure Databricks.

Instead of adding users and groups manual to your Azure Databricks environment, you can also sync them automatically from your Azure Active Directory to your Azure Databricks account with SCIM.. This is one of the recommendations from Databricks.

Other advantages are:

  • Stream less onboarding of new employees or teams in Azure Databricks.
  • Users can be easily deleted from the Azure Databricks workspaces through the Azure Active Directory. This ensures a consistent offboarding process and prevents unauthorized users from accessing sensitive data.

Their are a couple of important requirements to have in place before we can start, you need to have or be:

  • Azure Databricks account with a Premium Plan.
  • Azure Databricks account admin to provision users to your Azure Databricks account using SCIM.
  • Azure Databricks workspace admin to provision users to an Azure Databricks workspace using SCIM.
  • Azure Active Directory account must be a Premium edition account to be able to provision groups.
  • Provisioning of users is available for all Azure Active Directory editions (including the Azure AD Free)

Blog Serie

This blog post series contains the following topics, which I will post in the next few days:

  1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
  2. Assign and Provision users and groups in the Enterprise Application(SCIM)
  3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
  4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
  5. Add Service Principals to your Azure Databricks account using the account console
  6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning

There are 2 different options to provision users and groups to Azure Databricks using Azure Active Directory (AAD) at the Azure Databricks account level or at the Azure Databricks workspace level. This post is related to the Azure Databricks Account Level.

Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning

Azure Databricks account level

Before you start login to the Azure Databricks account console.

Make sure that you’re an Azure Databricks account admin. If you’re not an account admin, check who is an account admin( you see this on the main page of the user Management option). Ask the Account admin to grant you access, they can do this by clicking on the account name.

Once you’re Account Admin, click on the left side, click on the user setting icon(red).

adb-user-settings

 

 

 

 

 

 

 

Click on User Provisioning and click on set-up user provisioning.

Enable-user-prv-add-adb

Create-token-scim

Copy the SCIM token and the Account SCIM URL and store them in an Azure Key Vault. We need these settings later to configure the Enterprise Application.

Configure the Enterprise Application

In the Azure portal, go to Azure Active Directory > Enterprise Applications.

Click on new application and search for the “Azure Databricks SCIM Provisioning Connector”

app-scim-adb

Click on the app:

app-scim-adb-create

Enter a Name for the application, I used Azure Databricks SCIM AzureDataBricksWestEurope

Click on Create and wait until the application is created.

app-scim-adb-configure

Click on Provisioning and set Provisioning Mode to Automatic.

app-scim-adb-configure-automatics

Set the Tenant URL to the Account SCIM URL that we saved earlier in our Key Vault.

Set Secret Token to the Azure Databricks SCIM token that we generated and saved earlier in our Key Vault.

Click on Test Connection so see if everything is configured correctly.

In my next blog I will explain how to Assign and Provision users and groups in the Enterprise Application(SCIM)

Feel free to leave a comment

2 Comments

  1. Victor Yang

    hi, do you know if it can work with AD PIM Group? We are looking for to grant support user to access the production in short period time

    Reply
    • Erwin

      Hi Victor,

      I had this discussion with someone else today as well. PIM works, but the users are not deleted in Databricks because they are created as local users. I’ve put it on my to-do list to check it out. I’ll let you know when I’ve done that, but I have a full agenda at the moment

      Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

two × one =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Get control of data loads in Azure Synapse

Load Source data to DataLake There are several ways to extract data from a source in Azure Synapse Analytics or in Azure Data Factory. In this article I'm going to use a metadata-driven approach by using a control table in Azure SQL in which we configure the...

Azure DevOps and Azure Feature Pack for Integration Services

Azure Feature Pack for Integration ServicesAzure Blob Storage A great addition for SSIS is using extra connectors like  Azure Blob Storage or Azure Data Lake Store which are added by the Azure Feature Pack. This Pack needs to be installed on your local machine. Are...

Azure Synapse Pause and Resume SQL Pool

Pause or Resume your Dedicated SQL Pool in Azure Synapse Analytics Azure Synapse Analytics went GA in beginning of December 2020, with Azure Synapse we can now also create a Dedicated SQL Pool(formerly Azure SQL DW). Please read this document to learn what a Dedicated...

Azure Data Factory: How to assign a Customer Managed Key

Customer key With this new functionality you can add extra security to your Azure Data Factory environment. Where the data was first encrypted with a randomly generated key from Microsoft, you can now use the customer-managed key feature. With this Bring Your Own Key...

Updated Microsoft Purview Pricing and Applications

Microsoft Purview Pricing and introduction of Purview Applications The Microsoft Purview pricing page has been updated. Below I have listed most of the changes. The most important changes are the introduction of the Microsoft Purview Applications and the pricing of...

Azure Data Factory: Generate Pipeline from the new Template Gallery

Last week I mentioned that we could save a Pipeline to GIT. But today I found out that you can also create a Pipeline from a predefined Solution Template.Template Gallery These template will make it easier to start with Azure Data Factory and it will reduce...

SSMS 18.1: Schedule your SSIS Packages in Azure Data Factory

Schedule your SSIS Packages with SSMS in Azure Data Factory(ADF) This week SQL Server Management Studio version 18.1 was released, which can be downloaded from here. In version 18.1 the Database diagrams are back and from now on we can also schedule SSIS Packages in...

Exploring Azure Synapse Analytics Studio

Azure Synapse Workspace Settings In my previous article, I walked you through "how to create your Azure Synapse Analytics Workspace". It's now time to explore the brand new Synapse Studio. Most configuration and settings can be done through the Synapse Studio. In your...

Migrate Azure Storage to Azure Data Lake Gen2

Migrate Azure Storage to Storage Account with Azure Data Lake Gen2 capabilities Does it sometimes happen that you come across a Storage Account where the Hierarchical namespace is not enabled or that you still have a Storage Account V1? In the tutorial below I...

SSMS 18.xx: Creating your Azure Data Factory SSIS IR directly in SSMS

Creating your Azure Data Factory(ADF) SSIS IR in SSMS Since  version 18.0 we could see our Integration Catalog on Azure Instances directly. Yesterday I wrote an article how to Schedule your SSIS Packages in ADF, during writing that article I found out that you can...