Provision users and groups from AAD to Azure Databricks (part 1)

by Jan 17, 2023

Blog Serie: Provisioning identities from Azure Active Directory to Azure Databricks.

Instead of adding users and groups manual to your Azure Databricks environment, you can also sync them automatically from your Azure Active Directory to your Azure Databricks account with SCIM. This is one of the recommendations from Databricks.

Other advantages are:

  • Stream less onboarding of new employees or teams in Azure Databricks.
  • Users can be easily deleted from the Azure Databricks workspaces through the Azure Active Directory. This ensures a consistent offboarding process and prevents unauthorized users from accessing sensitive data.

Their are a couple of important requirements to have in place before we can start, you need to have or be:

  • Azure Databricks account with a Premium Plan.
  • Azure Databricks account admin to provision users to your Azure Databricks account using SCIM.
  • Azure Databricks workspace admin to provision users to an Azure Databricks workspace using SCIM.
  • Azure Active Directory account must be a Premium edition account to be able to provision groups.
  • Provisioning of users is available for all Azure Active Directory editions (including the Azure AD Free)

Blog Serie

This blog post series contains the following topics, which I will post in the next few days:

  1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
  2. Assign and Provision users and groups in the Enterprise Application(SCIM)
  3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
  4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
  5. Add Service Principals to your Azure Databricks account using the account console
  6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning

There are 2 different options to provision users and groups to Azure Databricks using Azure Active Directory (AAD) at the Azure Databricks account level or at the Azure Databricks workspace level. This post is related to the Azure Databricks Account Level.

Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning

Azure Databricks account level

Before you start login to the Azure Databricks account console.

Make sure that you’re an Azure Databricks account admin. If you’re not an account admin, check who is an account admin( you see this on the main page of the user Management option). Ask the Account admin to grant you access, they can do this by clicking on the account name.

Once you’re Account Admin, click on the left side, click on the user setting icon(red).

adb-user-settings

 

 

 

 

 

 

 

Click on User Provisioning and click on set-up user provisioning.

Enable-user-prv-add-adb

Create-token-scim

Copy the SCIM token and the Account SCIM URL and store them in an Azure Key Vault. We need these settings later to configure the Enterprise Application.

Configure the Enterprise Application

In the Azure portal, go to Azure Active Directory > Enterprise Applications.

Click on new application and search for the “Azure Databricks SCIM Provisioning Connector”

app-scim-adb

Click on the app:

app-scim-adb-create

Enter a Name for the application, I used Azure Databricks SCIM AzureDataBricksWestEurope

Click on Create and wait until the application is created.

app-scim-adb-configure

Click on Provisioning and set Provisioning Mode to Automatic.

app-scim-adb-configure-automatics

Set the Tenant URL to the Account SCIM URL that we saved earlier in our Key Vault.

Set Secret Token to the Azure Databricks SCIM token that we generated and saved earlier in our Key Vault.

Click on Test Connection so see if everything is configured correctly.

In my next blog I will explain how to Assign and Provision users and groups in the Enterprise Application(SCIM).

Feel free to leave a comment

2 Comments

  1. Victor Yang

    hi, do you know if it can work with AD PIM Group? We are looking for to grant support user to access the production in short period time

    Reply
    • Erwin

      Hi Victor,

      I had this discussion with someone else today as well. PIM works, but the users are not deleted in Databricks because they are created as local users. I’ve put it on my to-do list to check it out. I’ll let you know when I’ve done that, but I have a full agenda at the moment

      Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

four × 3 =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Azure Data Factory and Azure Synapse Analytics Naming Conventions

Naming Conventions More and more projects are using Azure Data Factory and Azure Synapse Analytics, the more important it is to apply a correct and standard naming convention. When using standard naming conventions you create recognizable results across different...

Provision users and groups from AAD to Azure Databricks (part 5)

In the previous blog, you assigned Users and Groups to an Azure Databricks Workspace. In this blog, you will learn how to assign Service Principals to an Azure Databricks Workspace and define the correct entitlements.

Azure Data Factory: How to assign a Customer Managed Key

Customer key With this new functionality you can add extra security to your Azure Data Factory environment. Where the data was first encrypted with a randomly generated key from Microsoft, you can now use the customer-managed key feature. With this Bring Your Own Key...

Azure Synapse Pause and Resume SQL Pool

Pause or Resume your Dedicated SQL Pool in Azure Synapse Analytics Azure Synapse Analytics went GA in beginning of December 2020, with Azure Synapse we can now also create a Dedicated SQL Pool(formerly Azure SQL DW). Please read this document to learn what a Dedicated...

Migrate Azure Storage to Azure Data Lake Gen2

Migrate Azure Storage to Storage Account with Azure Data Lake Gen2 capabilities Does it sometimes happen that you come across a Storage Account where the Hierarchical namespace is not enabled or that you still have a Storage Account V1? In the tutorial below I...

Azure Synapse Analyics costs analyis for Integration Runtime

AutoResolveIntegrationRuntime! The last few days I've been following some discussions on Twitter on using a separate Integration Runtime in Azure Synapse Analytics running in the selected region instead of auto-resolve. The AutoResolveIntegrationRuntime is...

Get control of data loads in Azure Synapse

Load Source data to DataLake There are several ways to extract data from a source in Azure Synapse Analytics or in Azure Data Factory. In this article I'm going to use a metadata-driven approach by using a control table in Azure SQL in which we configure the...

Service Healths in Azure

Creating Service Health Alerts in AzureAzure Portal In the Azure Portal go to Monitor – Service Health – Health alerts If you have created alerts before you will see them over here. Assuming you haven’t created an Alert before, we will start to create an Alert.1...

Scale SQL Database dynamically with Metadata

Scale SQL Database Dynamically with Metadata Use this template to scale up and down an Azure SQL Database in Azure Synapse Analytics or in Azure Data Factory. This article describes a solution template how you can Scale up or down a SQL Database within Azure Synapse...

SSMS 18.xx: Creating your Azure Data Factory SSIS IR directly in SSMS

Creating your Azure Data Factory(ADF) SSIS IR in SSMS Since  version 18.0 we could see our Integration Catalog on Azure Instances directly. Yesterday I wrote an article how to Schedule your SSIS Packages in ADF, during writing that article I found out that you can...