Understanding Connection Security Rules

Updated: January 20, 2009

Applies To: Windows 7, Windows Server 2008 R2

Connection security involves the authentication of two computers before they begin communications and the securing of information sent between two computers. Windows Firewall with Advanced Security uses Internet Protocol security (IPsec) to achieve connection security by using key exchange, authentication, data integrity, and, optionally, data encryption.

Connection security rules use IPsec to secure traffic while it crosses the network. You use connection security rules to specify that connections between two computers must be authenticated or encrypted. You might still have to create a firewall rule to allow network traffic protected by a connection security rule.

In this topic:

• IKE and AuthIP

• Key exchange

• Authentication

• Data protection

IKE and AuthIP

IPsec uses the Internet Key Exchange (IKE) and Authenticated IP (AuthIP) protocols to establish the protected connections between two peer computers. The resultant connection is called a security association (SA), and is represented on each computer as a list of the protocols, algorithms, and keys required to communicate with the peer computer.

IKE

IKE is an Internet standard, defined in RFC 2409 (https://go.microsoft.com/fwlink/?linkid=134810), that describes a mechanism to establish IPsec SAs. An SA is a combination of a mutually agreed upon policy and keys that define the security services and mechanisms that help protect communication between IPsec peers. Specifically, IKE combines the Internet Security Association and Key Management Protocol (ISAKMP) in RFC 2408 and the Oakley Key Determination Protocol in RFC 2412.

ISAKMP is a framework for negotiating protected communications that is independent of specific key exchange protocols, encryption and integrity algorithms, and authentication methods. ISAKMP includes facilities to identify and authenticate peers, manage SAs, and exchange key material.

IKE uses the Oakley protocol to generate secret key material for protected communications. IKE uses ISAKMP to negotiate SAs.

IPsec peers use IKE to perform main-mode and quick-mode negotiation. Main-mode negotiation creates the ISAKMP SA, which protects security negotiations. During main-mode negotiation, the initiator and the responder exchange a series of ISAKMP messages in order to negotiate the encryption and hashing algorithms for the ISAKMP SA, exchange key determination material, and identify and authenticate each other. In the quick-mode negotiation phase, the two IPsec SAs (one SA for inbound data and another SA for outbound data) are created, which protects the data sent between two IPsec peers. During quick-mode negotiation, the initiator and the responder exchange a series of encrypted ISAKMP messages to negotiate the encryption and hashing algorithms for both the inbound and outbound IPsec SAs. For more information about IKE-based main-mode and quick-mode negotiation, see IKE Negotiation for IPsec Security Associations (https://go.microsoft.com/fwlink/?linkid=134811).

IKE is supported on computers that are running Windows 2000 or later.

AuthIP

AuthIP is an enhanced version of IKE that offers additional flexibility with support for user-based authentication, authentication with multiple credentials, improved authentication method negotiation, and asymmetric authentication. Like IKE, AuthIP supports main-mode and quick-mode negotiation. AuthIP also supports extended mode, a part of IPsec peer negotiation during which a second round of authentication can be performed. Extended mode, which is optional, can be used for multiple authentications. For example, with extended mode you can perform separate computer-based and user-based authentications.

Multiple authentication support is part of IPsec enforcement for Network Access Protection (NAP). IPsec peers authenticate with computer credentials and use a health certificate to prove that they are compliant with system health requirements. For more information, see Network Access Protection (https://go.microsoft.com/fwlink/?linkid=111066). For more information about the features of AuthIP, see AuthIP in Windows Vista (https://go.microsoft.com/fwlink/?linkid=76867).

AuthIP is supported on computers running Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2.

Key exchange

To encrypt communications, two computers must be able to access the same shared key (session key) without sending the key across a network and compromising the secret.

The Diffie-Hellman key exchange is one of the oldest and most secure algorithms used for key exchange. The two parties publicly exchange keying information, which this version of Windows additionally protects with a hash function signature. Neither party ever exchanges the actual key; however, after their exchange of keying material, each can generate the identical shared key. A computer that eavesdrops on the conversation cannot generate the key using the information that is sent over the Internet.

The Diffie-Hellman keying material exchanged by the two parties is categorized into groups. Strong Diffie-Hellman groups combine stronger algorithms with longer key lengths to increase the degree of computational difficulty of attacking and compromising the key.

Windows Firewall with Advanced Security uses Diffie-Hellman to provide the keying material for all other encryption keys. Diffie-Hellman does not provide authentication. In the IPsec implementation for this version of Windows, identities are authenticated after the Diffie-Hellman exchange takes place, providing protection against man-in-the-middle attacks.

You can configure key exchange settings on the IPsec Settings tab of the Windows Firewall with Advanced Security Properties dialog box.

By default, when AuthIP is used instead of IKE and the authentication method specified can provide a shared secret, then that shared secret is used instead of performing a Diffie-Hellman exchange. If, because of regulatory requirements, you must force IPsec to use a DH exchange, you can specify Use Diffie-Hellman for enhanced security on the Customize Advanced Key Exchange Settings dialog box.

Dynamic rekeying

Windows Firewall with Advanced Security uses keys to protect data that is communicated between computers. It also controls how often a new key is generated during communication, using a method called dynamic rekeying. The communication is sent in blocks; each block of data is secured with a different key. This prevents an attacker who has obtained part of a communication and the corresponding session keys from obtaining the remainder of the communication. If no key lifetime values are configured, keys are regenerated automatically at default intervals.

Session key refresh limit

Repeated rekeying based on a session key can compromise the main mode shared secret. For this reason, there is a session key refresh limit.

For example, Alice on Computer A sends a message to Bob on Computer B, and then sends another message to Bob a few minutes later. The same session key material might be reused because an SA was recently established. If you want to limit the number of times this occurs, set the session key lifetime in sessions to a low number. If both a key lifetime in minutes and number of sessions are specified, the limit reached first causes the subsequent rekey. By default, Windows Firewall with Advanced Security does not specify a lifetime for the number of sessions. For more information, see Customize Advanced Key Exchange Settings.

Authentication

Authentication refers to the way in which identities are verified before communications begin. A computer can be configured to use several authentication methods. The methods are attempted by each peer in the order they are listed. The two peers must have at least one common authentication method or communication will fail. Creating multiple authentication methods increases the chance that a common method between two computers can be found.

Windows Server 2008 R2 and Windows 7 support several authentication methods, including the use of X.509 and Secure Sockets Layer (SSL) certificates, and the NTLM version 2 (NTLMv2) and Kerberos version 5 protocols supported by Active Directory Domain Services (AD DS). Both the computer and the user can be authenticated. The Kerberos v5 protocol is typically used in a domain isolation scenario to ensure that only computers that are members of the domain, and are thus considered trustworthy, can connect to protected computers. Certificates are typically used to authenticate in environments where AD DS-based Kerberos is not available, such as with computers that run operating systems other than Windows.

Data protection

Data protection includes both data integrity and data encryption protocols that are used to protect the data stream between the two computers after key exchange and authentication are complete.

• Data integrity uses message hashes to ensure that information is not being changed while in transit. Hash message authentication codes (HMAC) sign packets to verify that the information received is exactly the same as the information sent. This is called integrity and it is critical when data is exchanged over unsecured media.

  The hash is a cryptographic checksum that each peer must compute to verify the message. For example, the sending computer uses a hash function and shared key to compute the checksum for the message, including it with the packet. The receiving computer must perform the same hash function on the received message and shared key and compare it to the original (included in the packet from the sender). If the message has changed in transit, the hash values are different and the packet is rejected.

• Data encryption uses algorithms to conceal the information being transmitted. Because IPsec provides the ability to frequently regenerate keys during communication, the entire data set will not be compromised if one key is exposed.

For information about the data integrity and data encryption algorithms supported in Windows Server 2008 R2 and Windows 7, see IPsec Algorithms and Protocols Supported by Windows (https://go.microsoft.com/fwlink/?linkid=129230).