Get Microsoft Entra ID tokens for service principals

This article describes how a service principal defined in Microsoft Entra ID can also act as a principal on which authentication and authorization policies can be enforced in Azure Databricks. Service principals in an Azure Databricks workspace can have different fine-grained access control than regular users (user principals).

A service principal acts as a client role and uses the OAuth 2.0 client credentials flow to authorize access to Azure Databricks resources.

You can manage service principals in Databricks or by using the following procedure from the Azure portal.

You can also use the Microsoft Authentication Library (MSAL) to programmatically get a Microsoft Entra ID access token for a user instead of a service principal. See Get Microsoft Entra ID tokens for users by using MSAL.

Provision a service principal in Azure portal

1. Sign in to the Azure portal.

   Note

   The portal to use is different depending on whether your Microsoft Entra ID application runs in the Azure public cloud or in a national or sovereign cloud. For more information, see National clouds.

2. If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to provision the service principal.

3. In Search resources, services, and docs, search for and select Microsoft Entra ID.

4. Click + Add and select App registration.

5. For Name, enter a name for the application.

6. In the Supported account types section, select Accounts in this organizational directory only (Single tenant).

7. Click Register.

8. On the application page’s Overview page, in the Essentials section, copy the following values:

   • Application (client) ID
   • Directory (tenant) ID

   

9. To generate a client secret, within Manage, click Certificates & secrets.

   Note

   You use this client secret to generate Microsoft Entra ID tokens for authenticating Microsoft Entra ID service principals with Azure Databricks. To determine whether an Azure Databricks tool or SDK can use Microsoft Entra ID tokens, see the tool’s or SDK’s documentation.

10. On the Client secrets tab, click New client secret.

    

11. In the Add a client secret pane, for Description, enter a description for the client secret.

12. For Expires, select an expiry time period for the client secret, and then click Add.

13. Copy and store the client secret’s Value in a secure place, as this client secret is the password for your application.

Provision a service principal with the Azure CLI

See Create a Microsoft Entra ID (formerly Azure Active Directory) service principal with the Azure CLI.

Get a Microsoft Entra ID access token with the Microsoft identity platform REST API

To access the Databricks REST API with the service principal, you get and then use a Microsoft Entra ID access token for the service principal. For more information, see First case: Access token request with a shared secret.

1. Gather the following information:

   Parameter	Description
   Tenant ID	The Directory (tenant) ID for the related application registered in Microsoft Entra ID.
   Client ID	The Application (client) ID for the related application registered in Microsoft Entra ID.
   Client secret	The Value of the client secret for the related application registered in Microsoft Entra ID.

2. Use the preceding information along with curl to get the Microsoft Entra ID access token.

   curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
   https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token \
   -d 'client_id=<client-id>' \
   -d 'grant_type=client_credentials' \
   -d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
   -d 'client_secret=<client-secret>'
   

   Replace:

   • <tenant-id> with the registered application’s tenant ID.
   • <client-id> with the registered application’s client ID.
   • <client-secret> with the registered application’s client secret value.

   Do not change the value of the scope parameter. It represents the programmatic ID for Azure Databricks (2ff814a6-3304-4ab8-85cb-cd0e6f879c1d) along with the default scope (/.default, URL-encoded as %2f.default).

   For example:

   curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
   https://login.microsoftonline.com/a1bc2d34-5e67-8f89-01ab-c2345d6c78de/oauth2/v2.0/token \
   -d 'client_id=12a34b56-789c-0d12-e3fa-b456789c0123' \
   -d 'grant_type=client_credentials' \
   -d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
   -d 'client_secret=abc1D~Ef...2ghIJKlM3'
   

   The Microsoft Entra ID access token is in the access_token value within the output of the call.

Get a Microsoft Entra ID access token with the Azure CLI

Databricks does not recommend that you create Microsoft Entra ID tokens for Microsoft Entra ID service principals manually. This is because each Microsoft Entra ID token is short-lived, typically expiring within one hour. After this time, you must manually generate a replacement Microsoft Entra ID token. Instead, use one of the participating tools or SDKs that implement the Databricks client unified authentication standard. These tools and SDKs automatically generate and replace expired Microsoft Entra ID tokens for you, leveraging the following Databricks authentication types:

• Azure managed identities authentication
• MS Entra service principal authentication
• Azure CLI authentication

To access the Databricks REST API with the service principal, you get and then use a Microsoft Entra ID access token for the service principal.

1. Gather the following information:

   Parameter	Description
   Tenant ID	The Directory (tenant) ID for the related application registered in Microsoft Entra ID.
   Client ID	The Application (client) ID for the related application registered in Microsoft Entra ID.
   Client secret	The Value of the client secret for the related application registered in Microsoft Entra ID.

2. Get the correct Azure subscription ID for the Microsoft Entra ID service principal, if you do not already know this ID, by doing one of the following:

   • In your Azure Databricks workspace’s top navigation bar, click your username and then click Azure Portal. On the Azure Databricks workspace resource page that appears, click Overview in the sidebar. Then look for the Subscription ID field, which contains the subscription ID.

   • Use the Azure CLI to run the az databricks workspace list command, using the --query and -o or --output options to narrow down the results. Replace adb-0000000000000000.0.azuredatabricks.net with the name of your workspace instance, not including the https://. In this example, the 00000000-0000-0000-0000-000000000000 after /subscriptions/ in the output is the subscription ID.

     az databricks workspace list --query "[?workspaceUrl==\`adb-0000000000000000.0.azuredatabricks.net\`].{id:id}" -o tsv
     
     # /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.Databricks/workspaces/my-ws
     

     If the following message appears, you are signed in to the wrong tenant: The subscription of '<subscription-id>' doesn't exist in cloud 'AzureCloud'. To sign in to the correct tenant, you must run the az login command again, using the -t or --tenant option to specify the correct tenant ID.

     You can get the tenant ID for an Azure Databricks workspace by running the command curl -v <per-workspace-URL>/aad/auth and looking in the output < location: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000, where 00000000-0000-0000-0000-000000000000 is the tenant ID. See also Get subscription and tenant IDs in the Azure portal.

     az login -t <tenant-id>
     

3. After you have the correct Azure tenant ID, client ID, client secret, and subscription ID for your Microsoft Entra ID service principal, sign in to Azure by using the Azure CLI to run the az login command. Use the --service-principal option along with specifying the values for the parameters of Tenant ID (Directory (tenant) ID), Client ID (Application (client) ID), and Client secret (Value) for the related application registered in Microsoft Entra ID.

   az login \
   --service-principal \
   -t <Tenant-ID> \
   -u <Client-ID> \
   -p <Client-secret>
   

4. Confirm that you are signed in to the correct subscription for your signed-in Microsoft Entra ID service principal. To do this, run the az account set command, using the -s or --subscription option to specify the correct subscription ID.

   az account set -s <subscription-id>
   

5. Generate the Microsoft Entra ID access token for the signed-in Microsoft Entra ID service principal by running the az account get-access-token command. Use the --resource option to specify the unique resource ID for the Azure Databricks service, which is 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d. You can display just the Microsoft Entra ID token’s value in the output of the command by using the --query and -o or --output options.

   az account get-access-token \
   --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d \
   --query "accessToken" \
   -o tsv
   

Use the service principal’s Microsoft Entra ID access token to access the Databricks REST API

A service principal that is a Databricks user can authenticate to the Databricks REST API with Microsoft Entra ID tokens.

A service principal can also add itself as a workspace admin to a workspace if it has the Contributor or Owner role on the target workspace resource in Azure. If the service principal is a Contributor or Owner on the target workspace and you want to add it to a workspace using its Microsoft Entra ID token, go to Workspace-level API access for service principals that are not Azure Databricks users.

Otherwise, continue to API access for service principals that are Azure Databricks users and admins.

API access for service principals that are Azure Databricks users and admins

To complete this procedure, you must first add the service principal to the Azure Databricks account or workspace. You can add your service principal directly to your account, without granting it workspace access, by using the SCIM (Account) API.

You can add your service principal to a workspace by using the Service Principals API endpoint. This will also add the service principal to your Azure Databricks account. For example:

By using Databricks CLI version 0.205 or above (recommended):

Add the service principal:

databricks service-principals create --application-id 12a34b56-789c-0d12-e3fa-b456789c0123 --display-name "My Service Principal" -p <profile-name-that-references-calling-users-access-token>

Update the service principal’s workspace entitlements:

databricks service-principals patch 1234567890123456 --json @update-service-principal.json -p <profile-name-that-references-access-token>

update-service-principal.json:

{
  "Operations": [
    {
      "op": "add",
      "path": "entitlements",
      "value": {
        "value": "workspace-acccess",
        "value": "allow-cluster-create",
        "value": "databricks-sql-access"
      }
    }
  ],
  "schema": [
    "urn:ietf:params:scim:api:messages:2.0:PatchOp"
  ]
}

See Authentication for the Databricks CLI.

By using curl:

curl -X POST \
-H 'Authorization: Bearer <access-token>' \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/preview/scim/v2/ServicePrincipals \
-H 'Content-type: application/scim+json' \
-d @create-service-principal.json

create-service-principal.json:

{
  "displayName": "My Service Principal",
  "applicationId": "12a34b56-789c-0d12-e3fa-b456789c0123",
  "entitlements": [
    {
      "value": "allow-cluster-create"
    }
  ],
  "schemas": [
    "urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"
  ],
  "active": true
}

Skip ahead to Workspace-level API access for service principals that are not Azure Databricks users if either of the following are true:

• The Azure Databricks REST API that you want to call requires workspace admin access and the service principal is a member of the workspace, but does not currently have admin access to the workspace.
• The service principal has not already been added to the target Azure Databricks workspace.

1. Gather the following information.

   Parameter	Description
   Microsoft Entra ID access token	The Microsoft Entra ID access token returned from the request in Get a Microsoft Entra ID access token with the Microsoft identity platform REST API or Get a Microsoft Entra ID access token with the Azure CLI.

2. Use the Microsoft Entra ID access token along with curl to call the Databricks REST API. For example:

   By using Databricks CLI version 0.205 or above (recommended):

   databricks clusters list -p <profile-name-that-references-azure-ad-access-token>
   

   See Authentication for the Databricks CLI.

   By using curl:

   curl -X GET \
   -H 'Authorization: Bearer <access-token>' \
   https://<databricks-instance>/api/2.0/clusters/list
   

   Replace:

   • <access-token> with the Microsoft Entra ID access token.
   • <databricks-instance> with the per-workspace URL of your Azure Databricks deployment.
   • GET and /api/2.0/clusters/list with the appropriate HTTP operation and endpoint for the target Databricks REST API.

   For example:

   curl -X GET \
   -H 'Authorization: Bearer <access-token>' \
   https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/list
   

Workspace-level API access for service principals that are not Azure Databricks users

Follow this procedure if any of the following are true:

• The Azure Databricks REST API that you want to call requires workspace admin access and the service principal is a member of the workspace, but does not currently have admin access to the workspace.
• The service principal has not already been added to the target Azure Databricks workspace.

Requirements:

• The service principal requires the Contributor or Owner role on the target workspace resource in Azure.

1. Gather the following information:

   Parameter	Description
   Tenant ID	The Directory (tenant) ID for the related application registered in Microsoft Entra ID in Provision a service principal in Azure portal.
   Client ID	The Application (client) ID for the related application registered in Microsoft Entra ID.
   Client secret	The Value of the client secret for the related application registered in Microsoft Entra ID, which you created in Provision a service principal in Azure portal.
   Microsoft Entra ID access token	The Microsoft Entra ID access token returned from the request in Get a Microsoft Entra ID access token with the Microsoft identity platform REST API or Get a Microsoft Entra ID access token with the Azure CLI.
   Subscription ID	The ID (not the name) of the Azure subscription that is associated with the target Azure Databricks workspace. To get to this and the following information, see Open resources. To open the target resource, you can search on the Azure Databricks service type and any other information in Azure that you know about the target Azure Databricks workspace.
   Resource group name	The name of the Azure resource group that is associated with the target Azure Databricks workspace.
   Workspace name	The name in Azure of the target Azure Databricks workspace.

2. Use some of the preceding information along with curl to get a Microsoft Entra ID management endpoint access token.

   curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
   https://login.microsoftonline.com/<tenant-id>/oauth2/token \
   -d 'client_id=<client-id>' \
   -d 'grant_type=client_credentials' \
   -d 'resource=https%3A%2F%2Fmanagement.core.windows.net%2F' \
   -d 'client_secret=<client-secret>'
   

   Replace:

   • <tenant-id> with the registered application’s tenant ID.
   • <client-id> with the registered application’s client ID.
   • <client-secret> with the registered application’s client secret value.

   Do not change the value of the resource parameter. It represents the Microsoft Entra ID management endpoint (https://management.core.windows.net/, URL-encoded as https%3A%2F%2Fmanagement.core.windows.net%2F).

   For example:

   curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
   https://login.microsoftonline.com/a1bc2d34-5e67-8f89-01ab-c2345d6c78de/oauth2/token \
   -d 'client_id=12a34b56-789c-0d12-e3fa-b456789c0123' \
   -d 'grant_type=client_credentials' \
   -d 'resource=https%3A%2F%2Fmanagement.core.windows.net%2F' \
   -d 'client_secret=abc1D~Ef...2ghIJKlM3'
   

   The Microsoft Entra ID management endpoint access token is in the access_token value within the output of the call.

3. Use the Microsoft Entra ID management endpoint access token along with the rest of the preceding information and curl to call the Databricks REST API, for example:

    curl -X GET \
    -H 'Authorization: Bearer <access-token>' \
    -H 'X-Databricks-Azure-SP-Management-Token: <management-access-token>' \
    -H 'X-Databricks-Azure-Workspace-Resource-Id: /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Databricks/workspaces/<workspace-name>' \
    https://<databricks-instance>/api/2.0/clusters/list
   

   Replace:

   • <access-token> with the Microsoft Entra ID access token.

     • <management-access-token> with the Microsoft Entra ID management endpoint access token.
     • <subscription-id> with the ID of the subscription that is associated with the target Azure Databricks workspace.
     • <resource-group-name> with the name of the resource group that is associated with the target Azure Databricks workspace.
     • <workspace-name> with the name of the target Azure Databricks workspace.
     • <databricks-instance> with the per-workspace URL of your Azure Databricks deployment.
     • GET and /api/2.0/clusters/list with the appropriate HTTP operation and endpoint for the target Databricks REST API.

     For example:

     curl -X GET \
     -H 'Authorization:Bearer <access-token>' \
     -H 'X-Databricks-Azure-SP-Management-Token: abC1dE...ghIj23kl' \
     -H 'X-Databricks-Azure-Workspace-Resource-Id: /subscriptions/12a345...bcd6789e/resourceGroups/my-resource-group/providers/Microsoft.Databricks/workspaces/my-workspace' \
     https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/list
     

   Upon authenticating to the workspace, the service principal becomes an Azure Databricks workspace admin and no longer needs the Contributor or Owner role to access the workspace.