Background threads in ASP.net applications (Part 3 – threading side effects)

  • Article

In the final article of the series on the dangers of ASP.net background threading, to illustrate the dangers of such an architecture, I will start by introducing one more modification to the code of the sample application. This modification will be added to the code of the Global.asax that starts off the infinite loop thread that runs outside of the ASP.net thread pool. Here is the listing again, but notice the lines of code that have been added:

private void UpdateQoutes()

{
  //declare an array of strings to contain the stock
string[] stockSymbols = { "MSFT", "AAPL", "GOOG", "AMZN" };
string url = "https://finance.yahoo.com/webservice/v1/symbols/";

  string stockUrl; StockQuote quote; XDocument xdoc; int loopIteration = 1;
  do
  {
    //should we have a loop iteration that yields modulo 6 = 0, crash
if ((loopIteration % 5) == 0)
{
throw new Exception("Random crash");
} 

    //go through each of the symbols and run them
foreach (String stockSymbol in stockSymbols)
{
stockUrl = url + Server.UrlEncode(stockSymbol) + "/quote";

       xdoc = XDocument.Load(stockUrl);

       //create a new qoute
quote = new StockQuote();
quote.CompanyName = GetData(xdoc, "name");
quote.Price = Convert.ToDouble(GetData(xdoc, "price"));
quote.Volume = Convert.ToInt32(GetData(xdoc, "volume"));
quote.SymbolName = GetData(xdoc, "symbol");
quote.LastUpdate = DateTime.Now; 

       //save the symbol
Application[stockSymbol] = quote; 
     }

    //sleep for 100 Seconds
System.Threading.Thread.Sleep(100000);

    //increment the loop iteration
loopIteration = loopIteration + 1; 

   } while (true);

}

The first thing added is the creation of a private variable called loopIteration which we set to the value 1. Then inside the do {} while(true) infinite loop, I added a check to see if the result of the division of the value of the new variable by 5 is not zero – that is to say, each time the value of the variable is a multiple of 5 (5, 10, 15, etc) the if branch will be taken and an exception will be thrown by the code. If the if statement is not taken, the value of the variable will be incremented by one at the end of the loop.

If you attempt to run the sample in IIS, you will see that after a time, the application pool will crash, and you will get an event 5011 logged by WAS (the Windows Process Activation Service) inside the System application log:

    A process serving application pool 'MvcSample' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2952'. The data field contains the error number.

So what just happened?

When request execution happens inside IIS, ASP.net gives you a safety net. Almost all exceptions (except for a System.OutOfMemoryException, System.StackOverflowException and System.ExecutionEngineException) are caught by the runtime, if they are not caught by your application directly. Hence an exception like FileNotFoundException that would occur during a page execution (request 3 in the picture below) would wind up with a 500 error page being sent to the connecting user – who would see the ASP.net yellow screen of death, but the w3wp.exe process hosting the application would not crash. Hence all other threads inside the process could go on treating requests, and the only impacted person would be the user requesting that particular page.

For thread that we launched with the Application_Start() event handler, there is no associated context and the Thread itself is not part of the ASP.net thread pool. Hence, when an exception occurs, if it is not handled in the application code, it will be handed directly to the operating system to treat: the way Windows deals with such an exception is by crashing the w3wp.exe process – and all other threads and requests inside it. This is what happens when the throw statement is executed in the loop. If you are unlucky enough to send a request exactly when this happens, you will see a message indicating that "This page cannot be displayed" in your browser.

Another, more subtle danger is application domain unloading and reloading. The application domain is a data structure that loads the entire code of your ASP.net application inside the w3wp.exe process. Since we cannot unload .dlls (assemblies) in .Net, when a change is made to the application on disk – files are changed inside the bin folder, the web.config file is updated, the entire application domain has to unload and reload with a new copy of your application. You can read more about this process by following this link:

https://linqto.me/AppDomainUnload

Supposing that the application domain has to be unloaded, but that the thread we launched at the beginning of the application is still running and holding on to objects in the current instance of the application domain, the unload cannot complete. Hence, you can wind up with behavior of the following sort: you perform a change to your application, and you do not see the change when refreshing the application in the browser. This is because the old app domain cannot unload until the spawned thread finishes dealing with all the objects it was holding on to inside the old app domain – and since we are in an infinite loop, this will be never.

Also, when you attempt to recycle the application pool in IIS, you may see warnings from WAS in the event log indicating that the w3wp.exe process serving the pool took too long to shut down, and hence was killed. IIS has a mechanism by which, a w3wp.exe process serving an application pool is allowed 90 seconds to gracefully stop all threads and shut down when a recycle has to occur. Passed this time period (controlled by the shutdownTimeLimit - https://www.iis.net/configreference/system.applicationhost/applicationpools/add/processmodel - parameter in the configuration), WAS will issue a kill process and force shutdown the w3wp.exe process. In the case of the sample application, the looping thread will never relinquish control and will not allow the process to shut down even after 90 seconds, so WAS will have to proceed with a kill.

Conclusion:

It is never a good idea to spawn up background threads in your application. If you need to have something treated on another thread outside the thread that does request treatment, consider calling the Threadpool.QueueWorkItem (https://msdn.microsoft.com/en-us/library/system.threading.threadpool.queueuserworkitem%28v=vs.110%29.aspx ) method call – this will use a free thread from the ASP.net threadpool to execute the code you give it. Another possibility is to consider asynchronous execution using the new .Net syntax with async and await. I will be doing a series on this next year (2015) so stay tuned.

 

By Paul Cociuba
https://linqto.me/about/pcociuba

Comments

  • Anonymous
    December 16, 2014
    Paul, have you seen http://hangfire.io for executing background jobs in ASP.NET applications?

  • Anonymous
    December 16, 2014
    you mentioned at the end of the article to consider calling Threadpool.QueueWorkItem instead. What about Task.Run, shouldn't this be preferred?

  • Anonymous
    December 16, 2014
    In the past we have just run scheduled background jobs by attaching a handler to an expiring application cache item. after the handler finishes, we would update the cache item to set a new future expiry when the main handler finished. I always wondered if cache expiry handlers used something from the asp.net thread pool or if it allocated the thread from some place else. anyone know?

  • Anonymous
    December 17, 2014
    This article fails to mention the new QueueBackgroundWorkItem: blogs.msdn.com/.../queuebackgroundworkitem-to-reliably-schedule-and-run-long-background-process-in-asp-net.aspx Scott Hanselman discusses all options for running background tasks in ASP.NET apps here: www.hanselman.com/.../HowToRunBackgroundTasksInASPNET.aspx

  • Anonymous
    December 18, 2014
    @odinserj: No I did not see this before, it look like it is built on top of async await, and I would be interested to see how it is implemented under the covers.

  • Anonymous
    December 18, 2014
    @Andrei: task.run is another way of doing async await. When you await something, the piece of code is an anonymous method that retuns Task.

  • Anonymous
    December 19, 2014
    @Mike J: Mike, if you are using the following method: public void Insert( string key, Object value, CacheDependency dependencies, DateTime absoluteExpiration, TimeSpan slidingExpiration, CacheItemPriority priority, CacheItemRemovedCallback onRemoveCallback ) this will have an event fired upon the expiration of the cache item. As long as in the delegate you provide, you do not create your own threads, you are ok. There is actually a thread inside the w3wp.exe that handles the timer callbacks, and the  dispatches them to the next free thread in the ASP.net threadpool. So the answer to your question would be yes: your code gets run on the threadpool provided you follow the above.

  • Anonymous
    December 22, 2014
    Paul, please see the UnobservedTaskException event in TaskScheduler.  It allows you to set up a global exception hander for exceptions outside the ASP.Net thread pool and protects w3wp from crashing.  Typically you would use it to log thread exceptions to your error log.  See larslab.wordpress.com/.../unobserved-exceptions-from-tasks-causing-iis-to-crash which has a good summary.

  • Anonymous
    December 23, 2014
    @Mark - The problem with the article you cite is that we wind up trying to catch any and every possible exception - on all and any thread. This might not be what you want... what happens if you were to catch a StackOverflowException - you could not treat it since there was no more room on the stack to call new methods to treat the exception. I would caution against using this sort of approach. If you have problems with crashes, take a look at Debug Diag 2.0 - http://linqto.me/DebugDiag and use this tool to gather crash dumps. It has a very nice analysis feature that will hand you the crash stack.