Error Rerunning Failed Data Factory Pipeline from Failed Activity with a Secure Output Activity

Hello Connor Welham,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having "Error Rerunning Failed Data Factory Pipeline from Failed Activity with a Secure Output Activity" and according to your question.

There are ways to fix it, and you can rerun from the failed activity and successfully skipping over the run pipeline with a secure output in Azure Data Factory.

1. Since Azure Data Factory does not cache secure output, you need to rerun the pipeline from the activity that generates the secure output. In your case, rerun from the 'Get app id' activity. This ensures that the secure output is regenerated and available for subsequent activities.
   1. Determine which activity in the pipeline failed.
   2. Modify the pipeline to allow reruns from the point of failure. This might involve using a control flow activity like If Condition or Switch to check the status of the previous run.
   3. Make sure that the rerun logic includes steps to regenerate any secure outputs that were not produced due to the failure.
2. Implement checkpoints in your pipeline to save the state of the pipeline at various stages. This way, you can restart the pipeline from a specific checkpoint rather than from the beginning. But you need to plan carefully.
3. You can also use the 'On Failure' path in your activities to trigger specific actions, such as sending notifications or retrying the failed activity by implement error handling.
4. Then, if you can configure retry logic for activities that are prone to transient errors. This can help mitigate issues caused by temporary server errors without requiring a full pipeline rerun.

Check out the links for more steps: https://video2.skills-academy.com/en-us/azure/data-factory/tutorial-pipeline-return-value and https://azure.microsoft.com/en-us/blog/rerun-activities-inside-your-data-factory-pipelines

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Hello @Connor Welham

It seems that the error you are facing is due to the fact that Azure Data Factory does not cache secure output, which is causing the pipeline to skip over the execute pipeline activity that contains a secure output web activity in it, which has already run.

To fix this issue, you can try rerunning the pipeline from the activity that produced the secure output, which in your case is the Get app id activity.

Alternatively, you can rerun the entire pipeline run instead of just the failed activity.

If you want to rerun the pipeline from the failed activity, you can try modifying your pipeline to store the secure output in a persistent location, such as Azure Blob Storage or Azure Data Lake Storage, instead of using the secure output feature.

This will allow you to rerun the pipeline from the failed activity without encountering the error you are facing.

Another option is to use the activity-level retry feature in Azure Data Factory. You can set the retry count on the execute pipeline activity that contains the secure output web activity. During the pipeline execution, if this activity run fails, the next automatic retry will start from the last trial's failure point.

I hope this helps