Team Foundation Server: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

  • [アーティクル]

Updated 4/21/2010: The Windows patch to solve this problem has been released. It can be found here.

Recently, while dogfooding our internal pioneer server, a few members of our team started to notice this error message when downloading files from TFS:

C:\dev\code\somefile.txt: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

Luckily, we were able to reproduce the issue on a consistent enough basis to investigate the issue.  After a few fun days of client-side TFS logging, sever-side TFS logging, TFS activity logging, fiddler tracing, netmon tracing, procmon tracing and finally http.sys tracing we were able to determine that we were being affected by an http.sys bug that was introduced in Windows Server 2008 R2.

Are you experiencing the same issue that we were?

Let’s find out.  First of all, this bug is new to Windows Server 2008 R2.  If you are running any version of Windows Server 2003, Windows Server 2008 R1 RTM or Windows Server 2008 R1 SP1 then you are most likely not hitting the same bug that we were.  The issue tended to occur more often if the file that was being downloaded was relatively large (> 2MB) and rate of occurrence seemed to grow with the size of the file.  Finally, we were only able to reproduce the issue over slower networks (3 to 4 kb/sec file transfer rate). 

Great news! The bug has been fixed!

This bug has been fixed by the Windows team and they have released a QFE for it.  You can find the QFE here.  You will need to install in on all of your ATs.

The following lists the initial workarounds that we were able to come up with before this bug was fixed.  I am going to leave them up here for reference but the real solution should be to install the QFE mentioned above.

--

If you are still hitting the issue after setting up the proxy or you are unable to set up a proxy then you are not out of luck just yet.  First, a little more information about why this issue is happening.  Http.sys is the http protocol stack that IIS uses to perform http communication with clients.  It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold.  By default, that threshold is set to 240 kb/sec.  It turns out that there is a bug with this timer and it is causing connections to be prematurely killed.  We have found that lowering this threshold reduces the number of connections that are killed by the server. 

In order to lower the threshold follow these directions:

  1. Open the IIS Manager
  2. In the Connections pane, make sure the name of your AT is selected.
  3. In the middle pane (titled “<MachineName> Home”), make sure you are in the “Features View” (bottom) and scroll down to the Management section.
  4. Double-click the “Configuration Editor” icon.
  5. The middle pane should now have the title “Configuration Editor”.  In the Section pull down near the top, expand the system.applicationHost and select “webLimits”.
  6. You should now see a bunch of property value pairs, one of which is named “minBytesPerSecond”.  Its value is most like 240.  You will want to lower this value for the workaround. 

In general, the lower the threshold is, the lower the chances are that you hit the issue above.  However, you may not want to lower the setting too much because then connections that should be killed by this timer will not be appropriately cleaned up.  Play around with values to see what works best for you.

If you have any other questions, be sure to let us know.

Comments

  • Anonymous
    April 16, 2010
    Same issue happens if you have installed TFS 2010 on VMWare with Windows Server 2008(not R2) And changing the "minBytesPerSecond" value or other network realated solutions do not solve the problem... We have solved the problem, setting up WSS later after the creating the TFS 2010 project... When creating a new TFS 2010 project, do not set anything about TFS project's portal...After you created the project, set a new WSS site, and assing it to the TFS project...

  • Anonymous
    November 19, 2010
    You mention that " You will need to install in on all of your ATs."   What are ATs?

  • Anonymous
    December 24, 2010
    > What are ATs? Application Tier. TFS consists of a Application tier and a Database Tier. If your unfamiliar with the term, then you probably have just a single server installation, which means both the AT and DT are on the same server.

  • Anonymous
    March 15, 2011
    We've applied the HotFix and have tried the workaround , neither worked

  • Anonymous
    March 20, 2011
    Same issue here. Win2008 R2 SP1 with  TFS 2010 SP1. Remote clients are still hitting this error. Hotfix is not for SP1 Version of R2 and Workaround does not help.

  • Anonymous
    March 23, 2011
    Looks like we may also have this issue (Win 2008 R2 + TFS 2010 SP1). Is a fix included in Windows 2008 R2 SP1?

  • Anonymous
    March 25, 2011
    We have still this issue with installed SP1 and all updates.

  • Anonymous
    April 20, 2011
    Same issue here after SP1 and all updates and workarounds. :(

  • Anonymous
    December 22, 2011
    Even after applying the update we are experiencing this still

  • Anonymous
    September 30, 2012
    Same issue here, but only on our build server.  It is Windows Server 2008 Standard Edition SP2, on MS Hyper-V.

  • Anonymous
    March 20, 2013
    Getting the same issue everyone else is even applying the hotfix and the workaround.  When is Microsoft going to fix the issue???????

  • Anonymous
    September 23, 2013
    Do we need to install it on the TFS Server machine or the Build machine?

  • Anonymous
    July 28, 2014
    windows server 2008 R2 SP1+EXCHANGE 2013 CU1 DAG also has this issue.