Collect your Apache Spark applications logs and metrics using Azure Event Hubs (preview)

The Fabric Apache Spark diagnostic emitter extension is a library that enables Apache Spark applications to emit logs, event logs, and metrics to various destinations, including Azure Log Analytics, Azure Storage, and Azure Event Hubs.

In this tutorial, you learn how to use the Fabric Apache Spark diagnostic emitter extension to send Apache Spark application logs, event logs, and metrics to your Azure Event Hubs.

Collect logs and metrics to Azure Event Hubs

Step 1: Create an Azure Event Hubs Instance

To collect diagnostic logs and metrics, you can use an existing Azure Event Hubs instance. If you don't have one, you can create an event hub.

Step 2: Create a Fabric Environment Artifact with Apache Spark Configuration

Option 1: Configure with Azure Event Hubs Connection String

1. Create a Fabric Environment Artifact in Fabric

2. Add the following Spark properties with the appropriate values to the environment artifact, or select Add from .yml in the ribbon to download the sample yaml file which already containing the following properties.

   spark.synapse.diagnostic.emitters: MyEventHub
   spark.synapse.diagnostic.emitter.MyEventHub.type: "AzureEventHub"
   spark.synapse.diagnostic.emitter.MyEventHub.categories: "Log,EventLog,Metrics"
   spark.synapse.diagnostic.emitter.MyEventHub.secret: <connection-string>
   spark.fabric.pools.skipStarterPools: "true" //Add this Spark property when using the default pool.
   

   Fill in the <connection-string> parameters in the configuration file. For more information, see Azure Event Hubs configurations.

Option 2: Configure with Azure Key Vault

To configure Azure Key Vault for storing the workspace key:

1. Create and go to your key vault in the Azure portal.

2. On the settings page for the key vault, select Secrets, then Generate/Import.

3. On the Create a secret screen, choose the following values:

   • Name: Enter a name for the secret.
   • Value: Enter the <connection-string> for the secret.
   • Leave the other values to their defaults. Then select Create.

4. Create a Fabric Environment Artifact in Fabric.

5. Add the following Spark properties. Or select Add from .yml on the ribbon to download the sample yaml file, which includes following Spark properties.

   spark.synapse.diagnostic.emitters: MyEventHub
   spark.synapse.diagnostic.emitter.MyEventHub.type: "AzureEventHub"
   spark.synapse.diagnostic.emitter.MyEventHub.categories: "Log,EventLog,Metrics"
   spark.synapse.diagnostic.emitter.MyEventHub.secret.keyVault: <AZURE_KEY_VAULT_NAME>
   spark.synapse.diagnostic.emitter.MyEventHub.secret.keyVault.secretName: <AZURE_KEY_VAULT_SECRET_KEY_NAME>
   spark.fabric.pools.skipStarterPools: "true" //Add this Spark property when using the default pool.
   

   Fill in the following parameters in the configuration file: <AZURE_KEY_VAULT_NAME>, <AZURE_KEY_VAULT_SECRET_KEY_NAME>. For more details on these parameters, refer to Azure Event Hubs configurations.

6. Save and publish changes.

Step 3: Attach the Environment Artifact to Notebooks or Spark Job Definitions, or Set It as the Workspace Default

To attach the environment to Notebooks or Spark job definitions:

1. Navigate to the specific notebook or Spark job definition in Fabric.
2. Select the Environment menu on the Home tab and select the environment with the configured diagnostics Spark properties.
3. The configuration is applied when you start a Spark session.

To set the environment as the workspace default:

1. Navigate to Workspace Settings in Fabric.
2. Find the Spark settings in your Workspace settings (Workspace setting -> Data Engineering/Science -> Spark settings).
3. Select Environment tab and choose the environment with diagnostics spark properties configured, and click Save.

Available configurations

Log data sample

Here's a sample log record in JSON format:

{
  "timestamp": "2024-09-06T03:09:37.235Z",
  "category": "Log|EventLog|Metrics",
  "fabricLivyId": "<fabric-livy-id>",
  "applicationId": "<application-id>",
  "applicationName": "<application-name>",
  "executorId": "<driver-or-executor-id>",
  "fabricTenantId": "<my-fabric-tenant-id>",
  "capacityId": "<my-fabric-capacity-id>",
  "artifactType": "SynapseNotebook|SparkJobDefinition",
  "artifactId": "<my-fabric-artifact-id>",
  "fabricWorkspaceId": "<my-fabric-workspace-id>",
  "fabricEnvId": "<my-fabric-environment-id>",
  "executorMin": "<executor-min>",
  "executorMax": "<executor-max>",
  "isHighConcurrencyEnabled": "true|false",
  "properties": {
    // The message properties of logs, events and metrics.
    "timestamp": "2024-09-06T03:09:37.235Z",
    "message": "Initialized BlockManager: BlockManagerId(1, vm-04b22223, 34319, None)",
    "logger_name": "org.apache.spark.storage.BlockManager",
    "level": "INFO",
    "thread_name": "dispatcher-Executor"
    //...
  }
}

Fabric workspaces with Managed virtual network

Create a managed private endpoint for the target Azure Event Hubs. For detailed instructions, refer to Create and use managed private endpoints in Microsoft Fabric - Microsoft Fabric.

Once the managed private endpoint is approved, users can begin emitting logs and metrics to the target Azure Event Hubs.

Next steps

• Create Apache Spark job definition
• Create, configure, and use an environment in Microsoft Fabric
• Create and use managed private endpoints in Microsoft Fabric
• Develop, execute, and manage Microsoft Fabric notebooks
• Monitor Spark Applications