ADF - Capture the error message when you manually cancel the ADF pipeline run.

Finn 1 Reputation point
2020-11-11T00:40:48.343+00:00

I am trying to build an error logging mechanism in Azure Data Factory to capture all the errors into a SQL table.

This works fine if a pipeline activity failed itself, I have a stored procedure connecting to the activity (when failed) to write the ADF error message <@activity('activity').error.message> for a specific runid <@pipeline().RunId> into a table.

Another scenario is that the pipeline could be manually canceled. I was wondering if there is a way to get and log the 'cancel' status into a database table? Because a job can be canceled at any stage, there is no way I can put an error catching in place.

Many thanks for your help, any hints are most welcome.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. MartinJaffer-MSFT 26,081 Reputation points
    2020-11-11T19:12:38.95+00:00

    Hello @Finn and welcome to Microsoft Q&A.

    If my understanding is correct, you want to capture that a pipeline was cancelled, (not some other error that happens in a run that gets cancelled).

    You do raise a good point, that the act of cancelling a pipeline run precludes the ability to log that cancel via a pipeline activity.

    So I have thought up another way to monitor the pipeline.

    In the pipeline, make the first aactivity be a stored procedure which writes to your SQL the pipeline run ID ( @pipeline().RunId ). Then have your SQL server on a timed delay, query the Data Factory REST API to get the pipeline run's status by pipeline run ID. This should get you whether the pipeline is running, succeeded, cancelled, or failed.

    This way, as long as the initial activity succeeds in sending the run id to your SQL, you should always be able to track it.

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.