Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning

In one of my previous blogs, I explained how to setup the Enterprise Application for Azure Databricks account level  provisioning. This blog is related to the Azure Databricks Workspace Level. This is a slightly different setup hence a separate blog.

Preview

Provision identities to your Azure Databricks workspace is still in preview

Steps to configure

Login to the Azure Databricks Workspace as a workspace admin.

Generate a personal acces token and store it in a save location(Azure Key Vault)

Configure the Enterprise Application

In the Azure portal, go to Azure Active Directory > Enterprise Applications.

Click on new application and search for the "Azure Databricks SCIM Provisioning Connector"

Click on the app:

Enter a Name for the application, I used Azure Databricks SCIM MyAzureDatabricksWorkspace

Click on Create and wait until the application is created.

Click on Provisioning and set Provisioning Mode to Automatic.

Set the Tenant URL to https://<databricks-instance>/api/2.0/preview/scim

Set Secret Token to the Azure Databricks token that we generated and saved earlier in our Key Vault.

Click on Test Connection so see if everything is configured correctly.

You can learn how to assign and sync users in the following blogs as this is a similar approach to setting up at the account level.

1. Assign and Provision users and groups in the Enterprise Application(SCIM)
2. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements

Add Service principal to your Azure Databricks workspace

As a security best practice, Databricks recommends using an Azure AD service principal and its Azure AD token instead of your Azure Databricks user or your Databricks personal access token for your workspace user to give CI/CD platforms access to Azure Databricks resources. More details can be found in the following link Service principals for CI/CD

To add a service principal using Postman:

1. Create a new HTTP request (File > New > HTTP Request).
2. In the HTTP verb drop-down list, select POST.
3. Enter in the request URL   https://DATABRICKS ID.azuredatabricks.net/api/2.0/preview/scim/v2/ServicePrincipals
4. On the Authorization tab, in the Type list, select Bearer Token.
5. For Token, enter your Databricks personal access token for your workspace user.
6. On the Headers tab, add the Key and Value pair of Content-Type and application/scim+json
7. On the Body tab, select raw and JSON.
8. add
   {

     "displayName": "sp-edk-databricks-dvlm-deployment",

     "applicationId": "xxxxx",

     "entitlements": [

       {

         "value": "allow-cluster-create"

       }

     ],

     "schemas": [

       "urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"

     ],

     "active": true

   }

Click on send and the Service Principal is added to the Databricks Workspace and is ready for further usage. You can find the Service Principals in the admin console, groups, users.

When you use Postman I advise you to work with Environments and variables, this makes the reusing of scripts a lot easier. In this blog I have not done that for simplicity.

This was my last blog in the series, I hope you enjoyed reading these blogs. A summary of these blogs can be found below. If there are any questions or ambiguities, I would of course be happy to hear and answer them.

Other Blog post in this serie:

1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
2. Assign and Provision users and groups in the Enterprise Application(SCIM)
3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
5. Add Service Principals to your Azure Databricks account using the account console
6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning

Add Service principals to your Azure Databricks account using the account console

In the previous blog, you assigned Users and Groups to an Azure Databricks Workspace. In this blog, you will learn how to assign Service Principals to an Azure Databricks Workspace and define the correct entitlements.

As a security best practice, Databricks recommends using an Azure AD service principal and its Azure AD token instead of your Azure Databricks user or your Databricks personal access token for your workspace user to give CI/CD platforms access to Azure Databricks resources. More details can be found in the following link Service principals for CI/CD

To add a service principal to the account using the account console:

1. As an account admin, log in to the account console.
2. Click User management.
3. On the Service principals tab, click Add service principal. Click here to create a new service principal.
4. Enter a name for the service principal.
5. Under UUID, enter the Application (client) ID for the service principal.
6. Click Add.

Assign Service Principal to Azure Databricks Workspace

Log in to your Workspace, in case you're still logged in, in your account console, you can open the workspace directly from Data setting icon, on the left side.

Once the Workspace is open, select the admin console in the upper right corner.

Select Service Principals.

Add Service Principals.

Select the Service Principal you want to add one by one.

The Service Principal is now visible and you can assign the correct entitlements to the Service Principal.

Once the Service Principal has been added, the service principal will also be visible in your Azure Databricks account.

Workspaces

Click on the correct Workspace, permissions and you can see that the Service Principal is now visible here as well. If needed you can change the role from beying a regular user to a full admin

In my next  blog, you will how to configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning.

Other Blog post in this serie:

1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
2. Assign and Provision users and groups in the Enterprise Application(SCIM)
3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
5. Add Service Principals to your Azure Databricks account using the account console
6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning

Creating a metastore in your Azure Databricks account

In the previous blog you learned how to sync and assign users and groups to the Enterprise Application. In this blog, you will learn how to create a metastore and assign it to Azure Databricks workspaces to. This is a prerequisite to be able to assign users and groups to the Azure Databricks workspaces.

In the situation below we're creating a metastore that is accessed using a managed identity, which is recommended situation.

Before you can create a Metastore, you need to create an Azure Databricks access connector, which is a first-party Azure resource that lets you connect a system-assigned managed identity to an Azure Databricks account.

Requirements :

• You need to have an Azure Databricks account with a Premium Plan.
• You must be an Azure Databricks account admin.
• You must have an Azure Data Lake Storage Gen2 storage account(must be in the same region as your Azure Databricks Workspace).

Azure Databricks access connector

Log in to the Azure Portal as a Contributor or as an Owner of a resource group

Search for the Access Connector for Azure Databricks in the Marketplace and click on create.

Configure the Connector

• Subscription: Select the subscription where you want to create the access connector in.
• Resource group: This should be a resource group in the same region as the storage account that you will connect to.
• Name: The name of the connector.
• Region: Same region as the storage account that you will connect to.

Click Review + create.
When you see the Validation Passed message, click Create.

Grant the managed identity access to the storage account

• Log in to your Azure Data Lake Storage Gen2 account as an Owner or a user with the User Access Administrator Azure RBAC role on the storage account.
• Go to Access Control (IAM), click + Add, and select Add role assignment.
• Select the Storage Blob Data Contributor role and click Next.
• Under Assign access to, select Managed identity.
• Click +Select Members, and select Access connector for Azure Databricks.
• Search for your connector name, select it, and click Review and Assign.

Create the Metastore

Login to the Azure Databricks account console.

Click on the left side, click on the data setting icon.

Click on the "Create a Metastore" button.

• Name for the metastore.
• Region where the metastore will be deployed., this must be the same region as the workspaces, storage and access connector.
• ADLS Gen 2 path: Enter the path to the storage container that you will use as root storage for the metastore.
• Access Connector ID: Enter the Azure Databricks access connector’s resource ID, can be found on the main page of the access connector.

Click on the Create to create the Metastore.

If you see the following error, you forgot to assign the managed identity access to the storage account. You can Force Create the metastore and assign the managed identity afterwards.

Assign Workspace to Metastore

The last steps is to assign the Workspaces to the Metastore. Click on the right side Assign to workspace.

You will only see all workspaces which have not been assigned earlier. Select the correct workspace and click on assign.

Enable the Unity Catalog and your workspaces are connected.

In my next blog I will explain how to Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements.

Other Blog post in this serie:

1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
2. Assign and Provision users and groups in the Enterprise Application(SCIM)
3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
5. Add Service Principals to your Azure Databricks account using the account console
6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning