Long Running Tasks in Logic Apps

  • [アーティクル]

It’s a common scenario I get asked about often: “I’ve written a custom API app to integrate with Logic Apps, but it has a lot of work it has to do. It seems that Logic Apps is timing out the connection before the work completed."  It’s a great question - what do you do when you need to complete a task that could take minutes, or hours, but fit within the constraints of a REST-based architecture? It’s clear to see the challenge here. By default Logic Apps will only wait about a minute before timing out a request. However there are very simple REST-based patterns you can follow to allow to you perform tasks that last much longer than a minute. And the best part? These patterns are supported natively from the engine, so you don’t have to worry about adding the async pattern logic into your workflows. Let’s jump into how to do it:

The Basics

When running a long step or task, the first thing you need to do is make sure the engine knows you haven’t timed out. You also need to communicate with the engine how it will know when you are finished with the task, and finally, you need to return relevant data to the engine so it can continue with the workflow. You can complete that via an API by following the flow below. These steps are from the point-of-view of the custom API:

  1. When a request is recieved, immediately return a response (before work is done). This response will be a 202 ACCEPTED response, letting the engine know you got the data, accepted the payload, and are now processing. The 202 response should contain the following headers:
    • location header (required). This is an absolute path to the URL Logic Apps can use to check the status of the job.
    • retry-after (optional, will default to 20). This is the number of seconds the engine should wait before polling the location header URL to check status.
  2. When a job status is checked, perform the following checks:
    • If the job is done: return a 200 OK response, with the response payload.
    • If the job is still processing: return another 202 ACCEPTED response, with the same headers as the initial response

This pattern allows you to run extremely long tasks within a thread of your custom API, but keep an active connection alive with the Logic Apps engine so it doesn’t timeout or continue before work is completed. When adding this into your Logic App, it’s important to note you do not need to check the status. As soon as the engine sees a 202 ACCEPTED response with a valid location header, it will honor the async pattern and continue to poll the location header until a non-202 is returned.

The Code

I created a sample on GitHub of an ASP.NET API that follows this pattern. For the sample I store state within a static dictionary, but in production you would want to use something more persistent like Azure Table Storage. Feel free to check it out and see it in action (by default will hold the step for 2 minutes).

The noteable pieces are in the initial response:

 new Thread(() => doWork(id)).Start(); //Start the thread of work, but continue on before it completes
 HttpResponseMessage responseMessage = Request.CreateResponse(HttpStatusCode.Accepted); 
 responseMessage.Headers.Add("location", String.Format("{0}://{1}/api/status/{2}", Request.RequestUri.Scheme, Request.RequestUri.Host, id)); //Where the engine will poll to check status
 responseMessage.Headers.Add("retry-after", "20"); //How many seconds it should wait (20 is default if not included)
 return responseMessage;

And the controller method to check the status of the job:

 //If the job is complete
 if(runningTasks.ContainsKey(id) && runningTasks[id])
 {
 runningTasks.Remove(id);
 return Request.CreateResponse(HttpStatusCode.OK, "Some data could be returned here");
 }
 //If the job is still running
 else if(runningTasks.ContainsKey(id))
 {
 HttpResponseMessage responseMessage = Request.CreateResponse(HttpStatusCode.Accepted);
 responseMessage.Headers.Add("location", String.Format("{0}://{1}/api/status/{2}", Request.RequestUri.Scheme, Request.RequestUri.Host, id)); //Where the engine will poll to check status
 responseMessage.Headers.Add("retry-after", "20");
 return responseMessage;
 }
 else
 {
 return Request.CreateErrorResponse(HttpStatusCode.BadRequest, "No job exists with the specified ID");
 }

Obviously these patterns can be emulated in any language, as long as the required headers/response codes are generated. Feel free to comment or email if you have any questions about applying this pattern within Logic Apps.

Comments

  • Anonymous
    May 25, 2017
    Hi,I have created a logic app to test your sample but I'm having some issues.The app has a request action as the first step, and then I have a Http action which calls the API URL.The Http action gets triggered, but it fails after 2 mintues (4 retries), although the API app is configured to have a delay of only 30 seconds, and the retry-after header is set to 1 second.Any idea why the Http action is failing? Do I have to use a different kind of actions, or may be different sequence of the logic app steps?
  • Anonymous
    August 20, 2017
    Under-rated blog post, and amazing that its taken this long for someone to comment. I havent tried to pattern yet, but in this scenario - IS there an absolute limit to how long LogicApps will keep waiting for the 200 return?Can see this as a way to implement a Human Task Manager within Azure