Pause and resume Azure Data Factory pipeline

Hello @jigsm and thank you for your question.

If I understand correctly, you have an application which the customer interacts with, and triggers the pipeline. After the pipeline starts, you want the customer to confirm the pipeline should continue. If the customer either declines or does not respond, the pipeline should stop at that point.

For Data Factory V2 there is not a "pause" for pipelines, however I have something better.
I say better, because with the below method, you do not need to manage pausing and resuming pipelines. It is part of the workflow.

Insert a Validation Activity in the pipeline at the point which you want to pause. The validation activity is used to wait for a dataset to be ready (exist, or be a certain size). Let the customer confirmation fulfill the validation activity requirements. For example, the validation activity looks for a blob by name "confirm/{id}". Initially this blob does not exist. If the customer clicks OK, then create the blob. The validation activity finds it and pipeline resumes execution.

The validation activity lets you set retry and timeout. This can handle both the case of customer clicking "no" and the case of customer not responding. By controlling the time between retry and total duration of activity, you can match the lifetime to that of your application UI.

If the validation activity does not find the blob after retrying, it eventually "times out". In pipeline logic, this is viewed by dependencies as an activity failure status. This can be used to stop the pipeline execution, or do other logic.