Add Service principals to your Azure Databricks account using the account console

In the previous blog, you assigned Users and Groups to an Azure Databricks Workspace. In this blog, you will learn how to assign Service Principals to an Azure Databricks Workspace and define the correct entitlements.

As a security best practice, Databricks recommends using an Azure AD service principal and its Azure AD token instead of your Azure Databricks user or your Databricks personal access token for your workspace user to give CI/CD platforms access to Azure Databricks resources. More details can be found in the following link Service principals for CI/CD

To add a service principal to the account using the account console:

1. As an account admin, log in to the account console.
2. Click User management.
3. On the Service principals tab, click Add service principal. Click here to create a new service principal.
4. Enter a name for the service principal.
5. Under UUID, enter the Application (client) ID for the service principal.
6. Click Add.

Assign Service Principal to Azure Databricks Workspace

Log in to your Workspace, in case you're still logged in, in your account console, you can open the workspace directly from Data setting icon, on the left side.

Once the Workspace is open, select the admin console in the upper right corner.

Select Service Principals.

Add Service Principals.

Select the Service Principal you want to add one by one.

The Service Principal is now visible and you can assign the correct entitlements to the Service Principal.

Once the Service Principal has been added, the service principal will also be visible in your Azure Databricks account.

Workspaces

Click on the correct Workspace, permissions and you can see that the Service Principal is now visible here as well. If needed you can change the role from beying a regular user to a full admin

In my next  blog, you will how to configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning.

Other Blog post in this serie:

1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
2. Assign and Provision users and groups in the Enterprise Application(SCIM)
3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
5. Add Service Principals to your Azure Databricks account using the account console
6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning

Creating a metastore in your Azure Databricks account

In the previous blog you learned how to sync and assign users and groups to the Enterprise Application. In this blog, you will learn how to create a metastore and assign it to Azure Databricks workspaces to. This is a prerequisite to be able to assign users and groups to the Azure Databricks workspaces.

In the situation below we're creating a metastore that is accessed using a managed identity, which is recommended situation.

Before you can create a Metastore, you need to create an Azure Databricks access connector, which is a first-party Azure resource that lets you connect a system-assigned managed identity to an Azure Databricks account.

Requirements :

• You need to have an Azure Databricks account with a Premium Plan.
• You must be an Azure Databricks account admin.
• You must have an Azure Data Lake Storage Gen2 storage account(must be in the same region as your Azure Databricks Workspace).

Azure Databricks access connector

Log in to the Azure Portal as a Contributor or as an Owner of a resource group

Search for the Access Connector for Azure Databricks in the Marketplace and click on create.

Configure the Connector

• Subscription: Select the subscription where you want to create the access connector in.
• Resource group: This should be a resource group in the same region as the storage account that you will connect to.
• Name: The name of the connector.
• Region: Same region as the storage account that you will connect to.

Click Review + create.
When you see the Validation Passed message, click Create.

Grant the managed identity access to the storage account

• Log in to your Azure Data Lake Storage Gen2 account as an Owner or a user with the User Access Administrator Azure RBAC role on the storage account.
• Go to Access Control (IAM), click + Add, and select Add role assignment.
• Select the Storage Blob Data Contributor role and click Next.
• Under Assign access to, select Managed identity.
• Click +Select Members, and select Access connector for Azure Databricks.
• Search for your connector name, select it, and click Review and Assign.

Create the Metastore

Login to the Azure Databricks account console.

Click on the left side, click on the data setting icon.

Click on the "Create a Metastore" button.

• Name for the metastore.
• Region where the metastore will be deployed., this must be the same region as the workspaces, storage and access connector.
• ADLS Gen 2 path: Enter the path to the storage container that you will use as root storage for the metastore.
• Access Connector ID: Enter the Azure Databricks access connector’s resource ID, can be found on the main page of the access connector.

Click on the Create to create the Metastore.

If you see the following error, you forgot to assign the managed identity access to the storage account. You can Force Create the metastore and assign the managed identity afterwards.

Assign Workspace to Metastore

The last steps is to assign the Workspaces to the Metastore. Click on the right side Assign to workspace.

You will only see all workspaces which have not been assigned earlier. Select the correct workspace and click on assign.

Enable the Unity Catalog and your workspaces are connected.

In my next blog I will explain how to Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements.

Other Blog post in this serie:

1. Configure the Enterprise Application(SCIM) for Azure Databricks Account Level provisioning
2. Assign and Provision users and groups in the Enterprise Application(SCIM)
3. Creating a metastore in your Azure Databricks account to assign an Azure Databricks Workspace
4. Assign Users and groups to an Azure Databricks Workspace and define the correct entitlements
5. Add Service Principals to your Azure Databricks account using the account console
6. Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning