Azure Data Factory Pipelines are not listing in Azure Metrics filters

Question

Hi Azure,

I have a requirement to visualize the count of the failed pipeline runs for the recently created pipeline. I wanted to achieve this using Azure Metrics. But When I was creating a Metric chart, I found the recently created pipeline was not listed in the filters. The below screenshots will help you understand the scenario better.

Metric Name: Failed Pipeline runs Metrics

Here is the screenshot that shows the newly created Pipeline with the name Pipeline3:

When I was creating a Metric Chart, I can see the existing pipelines but not the newly created pipeline(pipeline3) in the list:

I faced a similar issue in another pipeline named pipeline1 in the above screenshot when there were no runs available for the past 10 days. Once I triggered a run in the ADF portal, I could see pipline1 in the list.

I want to understand what is restricting them to be listed in the filter. And What is the Maximum number of days it would take for a pipeline to go unavailable when there were no runs?

Thanks,

Accepted Answer

Hi @Nishanth Prabhakaran ,

Thanks for using Microsoft Q&A forum and posting your query.

Below are my observations.

I want to understand what is restricting them to be listed in the filter

As per my testing, the pipeline names are listed in the drop down of Metrics filters only when there is an entry for that criteria. Nothing but if you are looking for failed runs of a particular pipeline then there should be at least one failed instance of that pipeline. In case if all the previous runs of that pipeline are success and no failures, then you are not seeing that pipeline name in the drop down list while creating the metrics. But once there is a failure instance recorded then you will start seeing that pipeline in drop down list.

I have also reached out to internal team to get additional information about this behavior and will keep you posted as soon as I have more info from the team.

Maximum number of days it would take for a pipeline to go unavailable when there were no runs?

Data Factory stores pipeline-run data for only 45 days. In case if you want to keep that data for a longer time, you use Azure Monitor. With Monitor, you can route diagnostic logs for analysis to multiple different targets.
- Storage Account: Save your diagnostic logs to a storage account for auditing or manual inspection. You can use the diagnostic settings to specify the retention time in days.
- Event Hub: Stream the logs to Azure Event Hubs. The logs become input to a partner service/custom analytics solution like Power BI.
- Log Analytics: Analyze the logs with Log Analytics. The Data Factory integration with Azure Monitor is useful in the following scenarios:
  a) You want to write complex queries on a rich set of metrics that are published by Data Factory to Monitor. You can create custom alerts on these queries via Monitor.
  b) You want to monitor across data factories. You can route data from multiple data factories to a single Monitor workspace.

Hope this info helps. Do let us know if you have further query.

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Share via

Azure Data Factory Pipelines are not listing in Azure Metrics filters

0 additional answers

Your answer