Create an Event Hubs data connection for Azure Data Explorer with SDKs

  • Article

Azure Data Explorer offers ingestion from Event Hubs, a big data streaming platform and event ingestion service. Event Hubs can process millions of events per second in near real time.

In this article, you connect to an event hub and ingest data into Azure Data Explorer. For an overview on ingesting from Event Hubs, see Azure Event Hubs data connection.

To learn how to create the connection in the Azure Data Explorer web UI, Azure portal, or with an ARM template, see Create an Event Hubs data connection.

For code samples based on previous SDK versions, see the archived article.

Prerequisites

Create an event hub data connection

In this section, you establish a connection between the event hub and your Azure Data Explorer table. As long as this connection is in place, data is transmitted from the event hub into your target table. If the event hub is moved to a different resource or subscription, you need to update or recreate the connection.

  1. Install the Microsoft.Azure.Management.Kusto NuGet package.

  2. Create a Microsoft Entra application principal to use for authentication. You need the directory (tenant) ID, application ID, and client secret.

  3. Run the following code.

    var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx"; //Directory (tenant) ID
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx"; //Application ID
var clientSecret = "PlaceholderClientSecret"; //Client Secret
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
var credentials = new ClientSecretCredential(tenantId, clientId, clientSecret);
var resourceManagementClient = new ArmClient(credentials, subscriptionId);
var resourceGroupName = "testrg";
//The cluster and database that are created as part of the Prerequisites
var clusterName = "mykustocluster";
var databaseName = "mykustodatabase";
var subscription = await resourceManagementClient.GetDefaultSubscriptionAsync();
var resourceGroup = (await subscription.GetResourceGroupAsync(resourceGroupName)).Value;
var cluster = (await resourceGroup.GetKustoClusterAsync(clusterName)).Value;
var database = (await cluster.GetKustoDatabaseAsync(databaseName)).Value;
var dataConnections = database.GetKustoDataConnections();
var eventHubConnectionName = "myeventhubconnect";
//The event hub that is created as part of the Prerequisites
var eventHubResourceId = new ResourceIdentifier("/subscriptions/<eventHubSubscriptionId>/resourceGroups/<eventHubResourceGroupName>/providers/Microsoft.EventHub/namespaces/<eventHubNamespaceName>/eventhubs/<eventHubName>");
var consumerGroup = "$Default";
var location = AzureLocation.CentralUS;
//The table and column mapping are created as part of the Prerequisites
var tableName = "StormEvents";
var mappingRuleName = "StormEvents_CSV_Mapping";
var dataFormat = KustoEventHubDataFormat.Csv;
var compression = EventHubMessagesCompressionType.None;
var databaseRouting = KustoDatabaseRouting.Multi;
var eventHubConnectionData = new KustoEventHubDataConnection
{
    EventHubResourceId = eventHubResourceId, ConsumerGroup = consumerGroup,
    Location = location, TableName = tableName, MappingRuleName = mappingRuleName,
    DataFormat = dataFormat, Compression = compression, DatabaseRouting = databaseRouting
};
await dataConnections.CreateOrUpdateAsync(WaitUntil.Completed, eventHubConnectionName, eventHubConnectionData);
    Setting Suggested value Field description
    tenantId xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx Your tenant ID. Also known as directory ID.
    subscriptionId xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx The subscription ID that you use for resource creation.
    clientId xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx The client ID of the application that can access resources in your tenant.
    clientSecret PlaceholderClientSecret The client secret of the application that can access resources in your tenant.
    resourceGroupName testrg The name of the resource group containing your cluster.
    clusterName mykustocluster The name of your cluster.
    databaseName mykustodatabase The name of the target database in your cluster.
    dataConnectionName myeventhubconnect The desired name of your data connection.
    tableName StormEvents The name of the target table in the target database.
    mappingRuleName StormEvents_CSV_Mapping The name of your column mapping related to the target table.
    dataFormat csv The data format of the message.
    eventHubResourceId Resource ID The resource ID of your event hub that holds the data for ingestion.
    consumerGroup $Default The consumer group of your event hub.
    location Central US The location of the data connection resource.
    compression gzip or None The type of data compression.
    databaseRouting Multi or Single The database routing for the connection. If you set the value to Single, the data connection is routed to a single database in the cluster as specified in the databaseName setting. If you set the value to Multi, you can override the default target database using the Database ingestion property. For more information, see Events routing.

Remove an event hub data connection

To remove the event hub connection, run the following command:

kustoManagementClient.DataConnections.Delete(resourceGroupName, clusterName, databaseName, dataConnectionName);

Related content