Diagnostics logs and metrics for Workflow Orchestration Manager

This article walks you through the steps to:

• Enable diagnostics logs and metrics for Workflow Orchestration Manager in Azure Data Factory.
• View logs and metrics.
• Run a query.
• Monitor metrics and set the alert system in directed acyclic graph (DAG) failure.

Prerequisites

You need an Azure subscription. If you don't have an Azure subscription, create a free Azure account before you begin.

Enable diagnostics logs and metrics for Workflow Orchestration Manager

1. Open your Data Factory resource and select Diagnostic settings on the leftmost pane. Then select Add diagnostic setting.

   

2. Fill out the Diagnostic settings name. Select the following categories for the Airflow logs:

   • Airflow task execution logs
   • Airflow worker logs
   • Airflow DAG processing logs
   • Airflow scheduler logs
   • Airflow web logs
   • If you select AllMetrics, various Data Factory metrics are made available for you to monitor or raise alerts on. These metrics include the metrics for Data Factory activity and the Workflow Orchestration Manager integration runtime, such as AirflowIntegrationRuntimeCpuUsage and AirflowIntegrationRuntimeMemory.

   

3. Under Destination details, select the Send to Log Analytics workspace checkbox.

   

4. Select Save.

View logs

1. After you add diagnostic settings, you can find them listed in the Diagnostic setting section. To access and view logs, select the Log Analytics workspace that you configured.

   

2. Under the section Maximize your Log Analytics experience, select View logs.

   

3. You're directed to your Log Analytics workspace where you can see that the tables you selected were imported into the workspace automatically.

   

Other useful links for the schema:

• Azure Monitor Logs reference - ADFAirflowSchedulerLogs | Microsoft Learn
• Azure Monitor Logs reference - ADFAirflowTaskLogs | Microsoft Learn
• Azure Monitor Logs reference - ADFAirflowWebLogs | Microsoft Learn
• Azure Monitor Logs reference - ADFAirflowWorkerLogs | Microsoft Learn
• Azure Monitor Logs reference - AirflowDagProcessingLogs | Microsoft Learn

Write a query

1. Let's start with the simplest query that returns all the records in ADFAirflowTaskLogs. You can double-click the table name to add it to a query window. You can also enter the table name directly in the window.

   

2. To narrow down your search results, such as filtering them based on a specific task ID, you can use the following query:

   ADFAirflowTaskLogs
   | where DagId == "<your_dag_id>"
   and TaskId == "<your_task_id>"
   

Similarly, you can create custom queries according to your needs by using any tables available in LogManagement.

For more information, see:

• Log Analytics tutorial
• Kusto Query Language (KQL) overview - Azure Data Explorer | Microsoft Learn

Monitor metrics

Data Factory offers comprehensive metrics for Airflow integration runtimes, allowing you to effectively monitor the performance of your Airflow integration runtime and establish alerting mechanisms as needed.

1. Open your Data Factory resource.

2. In the leftmost pane, under the Monitoring section, select Metrics.

   

3. Select the Scope > Metric Namespace > Metric you want to monitor.

   

4. Review the multiline chart that visualizes the Integration Runtime CPU Percentage and Integration Runtime Dag Bag Size.

   

5. You can set up an alert rule that triggers when your metrics meet specific conditions. For more information, see Overview of Azure Monitor alerts.

6. Select Save to dashboard after your chart is finished or else your chart disappears.

   

Airflow metrics

To see the metrics available for Workflow Orchestration Manager, view the Airflow metrics listed in the Supported metrics table.