SBS 2003 Troubleshooting: DNS server 'ReservedPort' fix needed on all servers
Problem
DNS server (source port randomisation) fix 953230 (MS08-037) causes other services (inc IPSEC) to fail to start
The update to Microsoft DNS client and server (MS08-037) (kb 953230) fixes the source-port randomisation flaw in the DNS spec.
DNS client security update (951748).
DNS server security update (951746).
The whole thing was compounded by MS08-067 (958644) which seemed to accentuate the issue especially on SBS boxes.
HOWEVER... this wreaks havoc with port based UDP services and clients that use ports above 1024.
In particular:
956188 (http://support.microsoft.com/kb/956188/) You experience issues with UDP-dependent network services after you install DNS Server service security update 953230 (MS08-037)
956189 (http://support.microsoft.com/kb/956189/) Some services may not start or may not work correctly on a computer that is running Windows SBS after you install the DNS Server security update 953230 (MS08-037)
956190 (http://support.microsoft.com/kb/956190/) DNS queries that are sent across a firewall do not use random source ports after you install security update 953230 (MS08-037)
From MS KB 953230
DNS Source Port randomisation
As a part of DNS Source Port randomisation, Microsoft has reserved ports to reduce the source port randomisation risk. The default size of the Socket Pool on Windows Server 2003 and down-level platforms is 2500. This size is configurable by modifying the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DNS\Parameters\SocketPoolSize registry subkey.
When the value of the SocketPoolSize registry entry is changed, the DNS service must restart for the changes go into effect. On DNS servers that have other services listening on UDP ports, it is helpful to prevent the DNS service from conflicting with the reserved ports of other services.
For more information about ReservedPorts keys, click the following article number to view the article in the Microsoft Knowledge Base:
812873 (http://support.microsoft.com/kb/812873/) How to reserve a range of ephemeral ports on a computer that is running Windows Server 2003 or Windows 2000 Server
MaxUserPort
On Windows Server 2003 and Windows 2000 Server, the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort registry subkey is defined as the maximum port up to which ports may be allocated for wildcard binds. The value of the MaxUserPort registry entry defines the dynamic port range.
For more information about the MaxUserPort registry entry, visit the following Microsoft Web site:
http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/58791.mspx?mfr=true (http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/58791.mspx?mfr=true)
What is the effective port range when the value of the MaxUserPort registry entry is set explicitly?
The MaxUserPort registry entry has a different meaning in Windows Server 2008, in Windows Vista, in Windows Server 2003, and in Windows 2000 Server.
In Windows Server 2003 or Windows 2000 Server, the value of the MaxUserPort registry entry defines the dynamic port range. The range starts from 1024 to MaxUserPort.
In Windows Server 2008 or Windows Vista, the value of the MaxUserPort registry entry signifies the number of dynamic ports. The range is from Start range (default is 49152) to Start range + MaxUserPort.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:
929851 (http://support.microsoft.com/kb/929851/) The default dynamic port range for TCP/IP has changed in Windows Vista and in Windows Server 2008
The following is the fix behavior on Windows Server 2003 and down-level platforms after you install the MS08-037 DNS update:
• If the value of the MaxUserPort registry entry is set, allocate ports randomly from the [1024, MaxUserPort] range.
• If the value of the MaxUserPort registry entry is not set, allocate ports randomly from the [49152, 65535] range
The FIX...
On systems with [any of (but not exclusive to)] Exchange, ISA, and SBS the MaxUserPort registry entry IS set and consequently the system is vulnerable to the DNS server or client randomly allocating port(s) required for other services that generally use the lower end of ports (1025 - 49151) [EG IPsec=4500].
Even then some client/server software may specifically use ports in the Ephemeral range (49152-65535) and be affected.
Because the DNS client/server allocates ports at random the effects and issues will be random. Thus rebooting a system will almost always resolve the particular issue seen, but then may cause others.
The October 2008 RPC fix MS08-067 (958644) seemed to compound the issue and make the instances of issues, especially with the IPsec service, more frequent.
I would surmise that the RPC fix may have somehow reduced the randomness of the DNS port allocation. In almost all instances of applying 958644 the IPsec service failed to start upon reboot (port 4500), but then started upon next reboot. This would indicate some sort of predictability to the port allocation in either the DNS server or client on SBS.
From hereon we need to add client exception ports to the (Multi-String) ReservedPorts registry entry in: (in w2k the entry is REG_Multi_SZ)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Ports must be entered in the xxxx-yyyy format (eg 3456-3459)
Single ports must be entered in the nnnn-nnnn format (eg 4000-4000)
On SBS the recommended port exclusion list is:
1433-1434 [MSSQL](more if you have other SQL instances on the system)
1745-1745 [ISA Server 2000 or ISA Server 2004]
1080-1080 [SOCKS]
1718-1720 [H.323 Gatekeeper (ISA 2000 only)]
3343-3343 (already in many systems)
1645-1646 [IAS](Radius)
1701-1701 [L2TP]
1812-1813 [IAS](Radius)
2883-2883 [AUTD]
4500-4500 [IPSEC]
1801-1801 [MSMQ](already in some systems)
4070-4071 (BES)
3500-3619 [ISA Server 2000 only]
Conclusion
In principle this SHOULD only be required when the entry for MaxUserPort has been set, Unless you need to isolate ports in the upper (ephemeral) range (49152-65535). But don't bet on it!
This affects all versions of: Windows 2000, Server 2003 and Windows XP.
This affects both client applications and server applications where the client/server needs to emit data or receive data on a specific UDP port between 1025 and 65535.
Windows Vista - the DNS randomisation fix IS applicable to this system, so random allocation of higher UDP ports will be occurring, however none of the issues, errors, or workarounds detail Vista as applicable or affected. It may be that the DNS client allocates the emitting port randomly on-the-fly and thus avoids ports that are open or allocated. However, theoretically there could be a collision on some port that DNS used that an application then needed. So I (personally) would not discount Vista from being affected and these effects to be VERY random and impossible to diagnose.