Deploy Azure Data Explorer cluster into your Virtual Network

This article explains the resources that are present when you deploy an Azure Data Explorer cluster into a custom Azure Virtual Network. This information will help you deploy a cluster into a subnet in your Virtual Network (VNet). For more information on Azure Virtual Networks, see What is Azure Virtual Network?

Azure Data Explorer supports deploying a cluster into a subnet in your Virtual Network (VNet). This capability enables you to:

• Enforce Network Security Group (NSG) rules on your Azure Data Explorer cluster traffic.
• Connect your on-premises network to Azure Data Explorer cluster's subnet.
• Secure your data connection sources (Event Hubs and Event Grid) with service endpoints.

Access your Azure Data Explorer cluster in your virtual network

You can access your Azure Data Explorer cluster using the following IP addresses for each service (engine and data management services):

• Private IP: Used for accessing the cluster inside the virtual network.
• Public IP: Used for accessing the cluster from outside the virtual network for management and monitoring, and as a source address for outbound connections started from the cluster.

The following DNS records are created to access the service:

• [clustername].[geo-region].kusto.windows.net (engine) ingest-[clustername].[geo-region].kusto.windows.net (data management) are mapped to the public IP for each service.

• private-[clustername].[geo-region].kusto.windows.net (engine) ingest-private-[clustername].[geo-region].kusto.windows.net\private-ingest-[clustername].[geo-region].kusto.windows.net (data management) are mapped to the private IP for each service.

Plan subnet size in your virtual network

The size of the subnet used to host an Azure Data Explorer cluster can't be altered after the subnet is deployed. In your virtual network, Azure Data Explorer uses one private IP address for each VM and two private IP addresses for the internal load balancers (engine and data management). Azure networking also uses five IP addresses for each subnet. Azure Data Explorer provisions two VMs for the data management service. Engine service VMs are provisioned per user configuration scale capacity.

The total number of IP addresses:

Service endpoints for connecting to Azure Data Explorer

Azure Service Endpoints enables you to secure your Azure multitenant resources to your virtual network. Deploying the cluster into your subnet allows you to set up data connections with Event Hubs or Event Grid while restricting the underlying resources for Azure Data Explorer subnet.

Private Endpoints

Private Endpoints allow private access to Azure resources (such as storage/event hub/Data Lake Gen 2), and use private IP from your Virtual Network, effectively bringing the resource into your virtual network. Create a private endpoint to resources used by data connections, such as event hub and storage, and external tables such as Storage, Data Lake Gen 2, and SQL Database from your virtual network to access the underlying resources privately.

Configure Network Security Group rules

NSGs give you the ability to control network access within a virtual network. You must configure NSGs for your Azure Data Explorer cluster to work in your virtual network.

Configure Network Security Group rules using subnet delegation

Subnet delegation is the default method for configuring Network Security Group rules for Azure Data Explorer clusters deployed into a subnet in your virtual network. When using subnet delegation, you must delegate the subnet to Microsoft.Kusto/clusters before creating the cluster in the subnet.

By enabling subnet delegation on the cluster's subnet, you enable the service to define its pre-conditions for deployment in the form of Network Intent Policies. When creating the cluster in the subnet, the NSG configurations mentioned in the following sections are automatically created for you.

Configure Network Security Group rules manually

Alternatively, you can manually configure your NSG. By default, deploying a cluster into a virtual network enforces subnet delegation for "Microsoft.Kusto/clusters" to be configured. Opting out of this requirement is possible using the Preview features pane.

Inbound NSG configuration

Outbound NSG configuration

The following sections list the relevant IP addresses for management and health monitoring.

Azure Data Explorer management IP addresses

Health monitoring addresses

ExpressRoute setup

Use ExpressRoute to connect on premises network to the Azure Virtual Network. A common setup is to advertise the default route (0.0.0.0/0) through the Border Gateway Protocol (BGP) session. This forces traffic coming out of the Virtual Network to be forwarded to the customer's premise network that may drop the traffic, causing outbound flows to break. To overcome this default, User Defined Route (UDR) (0.0.0.0/0) can be configured and next hop will be Internet. Since the UDR takes precedence over BGP, the traffic will be destined to the Internet.

Securing outbound traffic with a firewall

If you want to secure outbound traffic using Azure Firewall or any virtual appliance to limit domain names, the following Fully Qualified Domain Names (FQDN) must be allowed in the firewall.

prod.warmpath.msftcloudes.com:443
gcs.prod.monitoring.core.windows.net:443
production.diagnostics.monitoring.core.windows.net:443
graph.windows.net:443
graph.microsoft.com:443
*.login.microsoft.com :443
*.update.microsoft.com:443
login.live.com:443
wdcp.microsoft.com:443
login.microsoftonline.com:443
azureprofilerfrontdoor.cloudapp.net:443
*.core.windows.net:443
*.servicebus.windows.net:443,5671
shoebox2.metrics.nsatc.net:443
prod-dsts.dsts.core.windows.net:443
*.vault.azure.net
ocsp.msocsp.com:80
*.windowsupdate.com:80
ocsp.digicert.com:80
go.microsoft.com:80
dmd.metaservices.microsoft.com:80
www.msftconnecttest.com:80
crl.microsoft.com:80
www.microsoft.com:80
adl.windows.com:80
crl3.digicert.com:80

Configure the route table

You must configure the route table of your cluster's subnet with next hop Internet to prevent asymmetric routes issues.

Configure the route table using subnet delegation

We recommend using subnet delegation to configure the route table for your cluster's deployment, similarly to how it was done for NSG rules. By enabling subnet delegation on the cluster's subnet, you enable the service to configure and update the route table for you.

Configure the route table manually

Alternatively, you can manually configure the route table. By default, deploying a cluster into a virtual network enforces subnet delegation for "Microsoft.Kusto/clusters" to be configured. Opting out of this requirement is possible using the Preview features pane.

To manually configure the route table you must define it on the subnet. You need to add the management and health monitoring addresses with next hop Internet.

For example, for West US region, the following UDRs must be defined:

How to discover dependencies automatically

Azure Data Explorer provides an API that allows customers to discover all external outbound dependencies (FQDNs) programmatically. These outbound dependencies will allow customers to set up a Firewall at their end to allow management traffic through the dependent FQDNs. Customers can have these firewall appliances either in Azure or on-premises. The latter might cause additional latency and might impact the service performance. Service teams will need to test out this scenario to evaluate impact on the service performance.

The ARMClient is used to demonstrate the REST API using PowerShell.

1. Sign in with ARMClient

   armclient login
   

2. Invoke diagnose operation

   $subscriptionId = '<subscription id>'
   $clusterName = '<name of cluster>'
   $resourceGroupName = '<resource group name>'
   $apiversion = '2021-01-01'
   
   armclient get /subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.Kusto/clusters/$clusterName/OutboundNetworkDependenciesEndpoints?api-version=$apiversion
   

3. Check the response

   {
      "value":
      [
       ...
         {
           "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.Kusto/Clusters/<clusterName>/OutboundNetworkDependenciesEndpoints/AzureActiveDirectory",
           "name": "<clusterName>/AzureActiveDirectory",
           "type": "Microsoft.Kusto/Clusters/OutboundNetworkDependenciesEndpoints",
           "etag": "\"\"",
           "location": "<AzureRegion>",
           "properties": {
             "category": "Azure Active Directory",
             "endpoints": [
               {
                 "domainName": "login.microsoftonline.com",
                 "endpointDetails": [
                   {
                     "port": 443
                   }
                 ]
               },
               {
                 "domainName": "graph.windows.net",
                 "endpointDetails": [
                   {
                     "port": 443
                   }
                 ]
               }
             ],
             "provisioningState": "Succeeded"
           }
         },
         {
           "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.Kusto/Clusters/<clusterName>/OutboundNetworkDependenciesEndpoints/InternalTracing",
           "name": "<clustername>/InternalTracing",
           "type": "Microsoft.Kusto/Clusters/OutboundNetworkDependenciesEndpoints",
           "location": "Australia Central",
           "properties": {
             "category": "Internal Tracing",
             "endpoints": [
               {
                 "domainName": "ingest-<internalTracingCluster>.<region>.kusto.windows.net",
                 "endpointDetails": [
                   {
                     "port": 443,
                     "ipAddress": "25.24.23.22"
                   }
                 ]
               }
             ],
             "provisioningState": "Succeeded"
           }
       }
       ...
      ]
   }
   

The outbound dependencies cover categories such as Microsoft Entra ID, Azure Monitor, Certificate Authority, Azure Storage, and Internal Tracing. In each category, there's a list of domain names and ports that are needed to run the service. They can be used to programmatically configure the firewall appliance of choice.

Deploy Azure Data Explorer cluster into your virtual network using an Azure Resource Manager template

To deploy Azure Data Explorer cluster into your virtual network, use the Deploy Azure Data Explorer cluster into your virtual network Azure Resource Manager template.

This template creates the cluster, virtual network, subnet, network security group, and public IP addresses.

Known limitations

• Virtual network resources with deployed clusters don't support the move to a new resource group or subscription operation.
• Public IP address resources used for the cluster engine or the data management service don't support the move to a new resource group or subscription operation.
• It's not possible to use the "private-" DNS prefix of virtual network injected Azure Data Explorer clusters as part of your query