getting error " 502 proxy error. The requested resource is in use. (170)." on the upstream proxy TMG servers

  • [アーティクル]

This one is about a case, where Admin had configured Two TMG 2010 upstream proxy servers and he had two TMG 2010 downstream proxy servers. Users started reporting that they intermittently get error

" 502 proxy error. The requested resource is in use. (170)." while accessing internet.

At the time of issue, even if you try to access it from the Upstream TMG server itself, you will see this error on the browser. To avoid the problem ,Admin was bypassing these upstream TMG proxy servers. So it appeared to be a Load issue.

Troubleshooting and Data analysis

During the course of troubleshooting, I collected TMG data packager while doing repro of the issue i.e. when issue occurred. In the TMG traces I found following.

  

 ************************************************ 
ERROR:Cannot create more than 3000 concurrent inspection sessions. 
ERROR:spScanner->CreateSession failed, hr=0x800700aa(ERROR_BUSY) 
ERROR:CreateInspectionSession() failed, hr=0x800700aa(ERROR_BUSY) 
ERROR:Accumulate failed, hr=0x800700aa(ERROR_BUSY) 
ERROR:pRequestContext->OnClientSendRawData failed, hr=0x800700aa(ERROR_BUSY) 

Warning m_spProtocolHandler->GetExpectedSize failed, hr=0x80070015(ERROR_NOT_READY) 
Info Parsing response headers hasn't completed yet, cannot check for filename. 
ERROR:pRequestContext->OnClientSendRawData failed, hr=0x800700aa(ERROR_BUSY) 

ERROR:SendResponseHeaders failed 170(ERROR_BUSY) 
Info WPPISAPUBLIC:Returning error text "The requested resource is in use. " for error code 170(ERROR_BUSY) 
********************************************* 

 We can see from above trace that we are getting error

 "ERROR:Cannot create more than 3000 concurrent inspection sessions. " in the trace, which means we are hitting the maximum 

 threshold for concurrent inspection sessions. 

 Then as a follow through we get message that server was busy. or Requested resource is in use.

  

 After doing some research on this behavior I suggested following :

   

 1. To increase the default value of 3000 via this RAT flag in registry 
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RAT\Stingray\Debug\Web Filters

 

 \EMP_MAX_CONCURRENT_INSPECTION_SESSIONS 

2. To add another upstream server i.e. scale up the number of servers to share

  the Load further to avoid reaching this limit of concurrent inspection sessions. 

After we configured the reg key with a value of 5000 on the TMG servers, 

 we did not see issue occurring again.

 

 We can change this value upto our requirements but better idea would be to scale up the

  number of servers as we may have to keep doing this if user base would keep increasing.

 

 

  

Comments

  • Anonymous
    January 01, 2003
    sorry guys i do not get any alerts for the comments on my blog, i just saw this and replying

    *********************************************************
    So key if you don’t see then create them as seen in the path given infront of key below, Then create Dword “EMP_MAX_CONCURRENT_INSPECTION_SESSIONS” and set a value as per your requirement , if you notice my blog post scaling up is a better idea.

    Note Please take backup of your registry keys before any modifications


    Key: HKEY_LOCAL_MACHINESOFTWAREMicrosoftRATStingrayDebugWeb Filters
    Value: EMP_MAX_CONCURRENT_INSPECTION_SESSIONS
    Type: REG_DWORD
    Data: 4500 -5000(Decimal)

    Hope that helps
  • Anonymous
    January 29, 2014
    Hi. How to view TMG traces? It's a ISALOG.BIN?
  • Anonymous
    October 13, 2014
    About the registry below I need the right syntax for the reg key value.
    please advice.
    [HKEY_LOCAL_MACHINESOFTWAREMicrosoftRATStingrayDebugWeb FiltersEMP_MAX_CONCURRENT_INSPECTION_SESSIONS] "5000"=""
  • Anonymous
    March 31, 2015
    Hi, our TMG is sometime reporting the same issue and we don't use upstream server. Also, the registry path showing here is not there in our TMG. Please provide the correct registry path
  • Anonymous
    August 26, 2015
    Hi. What application did you use to view the isalog.bin file?
  • Anonymous
    March 30, 2016
    FYI.
    Isalog.bin is only readable by Microsoft.