Troubleshooting LDAP over SSL connectivity.

  • [アーティクル]

Hello All,

Today I would like to talk to you about troubleshooting LDAP over SSL connectivity issues. We will be covering LDAP over SSL basics, how Subject Alternate Name's (SAN) work, configuring Active Directory Application Mode (ADAM) for LDAP over SSL, and of course simple troubleshooting steps.

LDAP OVER SSL BASICS

In order to enable LDAP over SSL, the following server and client requirements must be met:

SERVER REQUIREMENTS

The server must have a certificate stored in the local machine store that meets the following criteria:

  • Certificate Contains the Server Authentication OID: 1.3.6.1.5.5.7.3.1.
  • The Subject name or the first name in the SAN must match the FQDN of the host machine.
  • The Certificate passes the chaining validation test.
  • The host machine account has access to the private key.
    Note: Typically ADAM runs under a domain account as opposed to the Local System account. In this scenario the domain account must have access to the private key. This will be covered later in the blog.

For an easy way to validate whether or not the machine has a valid certificate, we can run the following command:

Certutil -VerifyStore MY

The output will look similar to the following:

================ Certificate 0 ================
Serial Number: 4678576700000000000e
Issuer: CN=Contoso Issuing CA, DC=Contoso, DC=Com
Subject: CN=ServerName.Contoso.com
Certificate Template Name: Machine
Non-root Certificate
Template: Machine, Computer
Cert Hash(sha1): d9 14 d3 cc 54 e7 02 3e a3 99 e6 31 0c 46 3d 03 81 c0 a7 cf
Key Container = e08a8f744d85c46b5494c876e5a9c7c2_17012b00-a428-4a32-bb81-3d15f8bc3c10
Provider = Microsoft RSA SChannel Cryptographic Provider
Private key is NOT exportable
Encryption test passed
Verified Issuance Policies: None
Verified Application Policies:
1.3.6.1.5.5.7.3.2 Client Authentication
1.3.6.1.5.5.7.3.1 Server Authentication
Certificate is valid

Note: We can of course have multiple certificates in our certificate store. So the value "================ Certificate 0 ================" refers to the first certificate in the store as the index values are zero based.

We can break down the output as follows:

Subject, i.e. the name that we specify for our LDAP over SSL Connection:

Subject: EMPTY (DNS Name=ServerName.Contoso.com)

The following section lets us know that we have a valid private key:

Private key is NOT exportable
Encryption test passed

The following verifies the intended purpose of the certificate which is Server Authentication:

Verified Application Policies:
1.3.6.1.5.5.7.3.1 Server Authentication

The last section, verifies that the certificate is indeed valid. I.e. the certificate chains to a trusted issuer, is within the time validity period, and has not been revoked.

Certificate is valid

Now we can of course run into issues at it relates to certificate validation. These will fall primarily into one of two categories, issues with the private key and issues with certificate chaining. We will cover the private key first.

Private Key

A typical error message would be:

No Key Provider Information or Missing Stored Keyset

This problem is due to a missing private key. We can confirm this by looking for the following in the Certutil output:

Cert Hash(sha1): a5 79 2f 21 82 99 4d f2 31 83 00 81 2c 84 85 3c 20 b7 5e 08
No key provider information
Missing stored keyset

The normal cause of this problem is that the certificate request was generated on one machine and we have installed the certificate on a different machine.

When we generate a certificate request, the client generates a private key and signs the request with it. When we receive the certificate from the CA, we can verify that the certificate is based on the request that was generated by the client.

So the first step in resolving the issue is verifying which machine the certificate request was generated on. We can then go to that machine and run the following command to associate the certificate with private key container:

C:\>Certutil -RepairStore MY 0
================ Certificate 0 ================
Serial Number: 334205f9000000000022
Subject:
CN=MachineName.Contoso.com
Non-root Certificate
Cert Hash(sha1): a5 79 2f 21 82 99 4d f2 31 83 00 81 2c 84 85 3c 20 b7 5e 08
Key Container = 574d09d6-9ea4-4a64-9a2a-dc1dfabd97c9
Provider = Microsoft Enhanced Cryptographic Provider v1.0
Private key is exportable
Signature test passed
CertUtil: -repairstore command completed successfully.

We have now associated the certificate with the private key. If this command fails then it means that the private key was not located in the machine store. If we can't locate the private key container then we will need to request a new certificate. Also, if the private key is marked as exportable we can export the certificate to the appropriate machine. If not we need a new certificate.

Certificate Validation Errors

Certificate validation is the process of verifying that the information contained in the certificate is authentic and that the certificate can only be used for its intended purpose and that the certificate is trusted.

If we have a validation issue we will see one of the following errors at the very bottom of the Certutil output:

Example 1:

A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file. 0x800b0101 (-2146762495)
------------------------------------
Expired certificate

CertUtil: -verifystore command completed successfully.

Example 2

The certificate is revoked. 0x80092010 (-2146885616)
------------------------------------
Certificate is REVOKED
Leaf certificate is REVOKED (Reason=6)
CertUtil: -verify command completed successfully.

Example 3

A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. 0x800b0109 (-2146762487)
------------------------------------
Verifies against UNTRUSTED root

CertUtil: -verifystore command completed successfully.

Example 4

An internal certificate chaining error has occurred. 0x800b010a (-2146762486)
------------------------------------
419.3000.0: 0x800b010a (-2146762486)
Incomplete certificate chain
Cannot find certificate:
CN=Microsoft Corp Enterprise CA 1, DC=redmond, DC=corp, DC=microsoft, DC=com

Example 5

The revocation function was unable to check revocation for the certificate. 0x80092012 (-2146885614)
------------------------------------
Revocation check skipped -- no revocation information available
Certificate is valid

Example 1, "Expired certificate", simply means that the certificate is beyond its validity period. Now it's possible that the system clock has been changed to an invalid date. Changing it to the correct current time will resolve the issue. However, if this is not the case then we will need to get a new certificate. Also, please note that you cannot renew a certificate once it has expired.

Example 2 "Certificate is REVOKED", means that the certificate has been revoked and therefore a new certificate needs to be issued.

For examples 3 & 4 i.e. the "UNTRUSTED root" and "Revocation" errors, troubleshooting is a little more involved. The "UNTRUSTED root" error means that one of the certificates in the chain is missing from the "Intermediate Certification Authorities" container for Intermediate certificates or from the "Trusted Root Certification Authorities" for root certificates. The "Revocation" error means that either the CRL is not cached locally on the client or we are unable to download the CRL from one of the publication points.

Fortunately, troubleshooting these errors is straightforward. First we need to dump the certificate to a file. The syntax is as follows:

CertUtil -store CertificateStoreName CertId OutputFile

For Example:

Certutil -store My 0 ProbCert.cer

The next step would be to verify whether or not the certificate can access the Authority Information Access (AIA) and the Certificate Distribution Point (CDP). If we can get to at least one of the paths for each certificate in the chain, the validation test will pass. The output will look similar to the following:

Verified Issuance Policies: None
Verified Application Policies:
1.3.6.1.5.5.7.3.2 Client Authentication
1.3.6.1.5.5.7.3.1 Server Authentication
Leaf certificate revocation check passed
CertUtil: -verify command completed successfully.

If we are unable to access one of the paths, we will need to perform additional analysis of the output. In reading the output, we start at the top. Next we perform a search on the following text "CertContext". The line will look similar to the following:

CertContext[0][0]: dwInfoStatus=102 dwErrorStatus=0

In this section we will see the "Subject" of the certificate i.e. the end entity that the certificate was issued to:

Subject: CN=ComputerName.Contoso.com

We will also see the Issuer of the certificate I.e.

Issuer: CN=Contoso Issuing CA, DC=Contoso, DC=com

Our next step is to locate the following section:

---------------- Certificate AIA ----------------

This section lets us know where the Issuer's certificate is located and whether or not the client can access it. Please note that this section can contain multiple paths. The key however is that we only need to be able to access one of the paths. So in our test, the following output is fine:

---------------- Certificate AIA ----------------
Verified "Certificate (0)" Time: 0
[0.0]
ldap:///CN=Contoso%20Issuing%20CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=Contoso,DC=Com?cACertificate?base?objectClass=certificationAuthority

Failed "AIA" Time: 0
Error retrieving URL: Error Error response received from gateway 0x800701f6
(WIN32/HTTP: 502)

https://ContosoIssuingCA.Contoso.com/CertEnroll/CONTOSOISSUINGCA.Contoso.com\_Contoso%20Issuing%20CA.crt

As we can see, we are able to access the LDAP path to the AIA. However, we are unable to access the HTTP path. Again, since we can access one of the paths, the validation check passes. If we cannot get to either of the paths, then this will have to be resolved. In this case the proxy is blocking access to the HTTP paths.

Our next step is to perform a search for the following section:

---------------- Certificate CDP ----------------

The CDP path is the publication path to the certificate revocation list. The application performing the validation test checks to see whether or not the certificate has been revoked. Again, if we can access any of the paths, the validation test passes. So again in our test we can get to the LDAP paths so the validation test passes.

---------------- Certificate CDP ----------------
Verified "Base CRL (64)" Time: 0
[0.0]
ldap:///CN=Contoso%20Issuing%20CA,CN=CONTOSOISSUINGCA,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=Contoso,DC=Com?certificateRevocationList?base?objectClass=cRLDistributionPoint

Verified "Delta CRL (64)" Time: 0
[0.0.0]
ldap:///CN=Contoso%20Issuing%20CA,CN=CONTOSOISSUINGCA,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=Contoso,DC=Com?deltaRevocationList?base?objectClass=cRLDistributionPoint

Failed "CDP" Time: 0
Error retrieving URL: Error Error response received from gateway 0x800701f6
(WIN32/HTTP: 502)
[0.1.0] https://ContosoIssuingCA.Contoso.Com/CertEnroll/Contoso%20Issuing%20CA+.crl

Failed "CDP" Time: 0
Error retrieving URL: Error Error response received from gateway 0x800701f6
(WIN32/HTTP: 502)
https://ContosoIssuingCA.Contoso.Com/CertEnroll/Contoso%20Issuing%20CA.crl

The next section we need to look for is:

---------------- Base CRL CDP ----------------

This section is for Delta CRLs which may or may not be available depending whether or not Delta CRLs are published. The paths are as follows:

---------------- Base CRL CDP ----------------
OK "Delta CRL (65)" Time: 0
[0.0]
ldap:///CN=Contoso%20Issuing%20CA,CN=CONTOSOISSUINGCA,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=Contoso,DC=Com?deltaRevocationList?base?objectClass=cRLDistributionPoint

Failed "CDP" Time: 0
Error retrieving URL: Error Error response received from gateway 0x800701f6
(WIN32/HTTP: 502)
https://ContosoIssuingCA.Contoso.com/CertEnroll/Contoso%20Issuing%20CA+.crl

--------------------------------
CRL 64:

Again we can get to the LDAP path but not the HTTP path. However, one key difference here is that the certificate validation test may or may not fail if the client is unable to access either path. This is completely dependent upon the application making the CRL check. In the case of LDAP, this is not checked.

This completes the validation test for this level of the certificate chain. If the issuer of the certificate is a Root Certificate then this completes the validation process. However, if this is a subordinate certificate, then we will go to the next level of the validation test. The section will look similar to the following:

CertContext[0][1]: dwInfoStatus=102 dwErrorStatus=0
Issuer: CN=Contoso Policy CA, DC=Contoso, DC=Com
Subject: CN=Contoso Issuing CA, DC=Contoso, DC=Com

As we can see the "Subject" name at this level, was the "Issuer" of the first level. So now we are performing a validation test on the Issuing CA certificate. However the validation test is exactly the same. Look at each level and verify that we can access at least one path in the AIA section and one path in the CDP section.

Again after we finish this section, we get to the next level in the chain:

CertContext[0][2]: dwInfoStatus=102 dwErrorStatus=0
Issuer: CN=Contoso Root CA, DC=Contoso, DC=Com
Subject: CN=Contoso Policy CA, DC=Contoso, DC=Com

So our subject at this level is the issuer from the previous level. Again, we perform the validation test and we need to verify that we can access at least one path in the AIA section and one path in the CDP section.

Assuming that we have a three tier CA configuration, we finally get to the root.

Note: We know that we have reached the root because we have a self-signed certificate i.e. the subject name matches the issuer name.

CertContext[0][2]: dwInfoStatus=10c dwErrorStatus=0
Issuer: CN=Contoso Root CA, DC=Contoso, DC=Com
Subject: CN=Contoso Root CA, DC=Contoso, DC=Com
Serial: 341f8fdd3ffce6934ee3900117eaee4e
08 85 f3 de 95 3e ac d8 78 f7 3f a8 06 0a 0f 59 bf 39 5a f6
Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4)
Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8)
Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)
---------------- Certificate AIA ----------------
No URLs "None" Time: 0
---------------- Certificate CDP ----------------
Verified "Base CRL (19)" Time: 0
[0.0]
ldap:///CN=Contoso%20Root%20CA,CN=R2DOMDC1,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=Contoso,DC=Com?certificateRevocationList?base?objectClass=cRLDistributionPoint

Failed "CDP" Time: 0
Error retrieving URL: Error Error response received from gateway 0x800701f6
(WIN32/HTTP: 502)
https://r2domdc1.r2domain.jlc/CertEnroll/R2Domain%20Root%20CA.crl

A Windows Server 2003 Root CA does not chain the root since it's the final authority. However, by default we include the AIA and CDP paths. Therefore, for the Root CA this can be ignored even if we have errors.

Note: We can prevent the paths from appearing in the root by configuring a CAPolicy.inf file prior to issuing or renewing the root certificate. For more information, please see the following:

CAPolicy.inf Syntax
https://technet2.microsoft.com/windowsserver/en/library/25127c1f-4880-4764-85e8-226ce41588881033.mspx?mfr=true

So at this point we have validated the entire certificate chain. If we have errors we need to resolve them. Most of the errors will be one of the following:

  1. CA Cert or CRL file is missing from the publication location.
  2. We are unable to access the publication location due to permissions. We will most likely see an access denied error. We need to check both user and machine account permissions.
  3. We are unable to access the AIA or CRL due to proxy settings. Again, we need to check both user and machine proxy settings.

After fixing the errors, we need to re-run the Certutil -Verify command. If no errors are reported at the bottom of the output, then the certificate is valid.

ADAM Special Case

Validating the server certificate for an ADAM instance is exactly the same with one caveat. As stated earlier in the blog, typically ADAM runs under a domain account. This account will have to have access to the private key container.

To locate the private key container for a certificate, run the following command:

Certutil -V -Verifystore MY 0

Look for the following section in the output:

CERT_KEY_PROV_INFO_PROP_ID(2):
Key Container = e08a8f744d85c46b5494c876e5a9c7c2_17012b00-a428-4a32-bb81-3d15f8bc3c10
Provider = Microsoft RSA SChannel Cryptographic Provider
ProviderType = c
Flags = 60
KeySpec = 1

This key container value, "e08a8f744d85c46b5494c876e5a9c7c2_17012b00-a428-4a32-bb81-3d15f8bc3c10" is the private key container file. We can find this value under:

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

We need to verify that the ADAM service account has "Read" permissions on the container.

This concludes the section on validating the server certificate. Next we will look at the client requirements.

CLIENT REQUIREMENTS

Unlike the server, the client does not require a client certificate for making the LDAP over SSL connections. However, the client does have to trust the server certificate and has to be able to verify the server's revocation status. To verify whether or not the client trust the server certificate we need to export the server certificate to the client.

To export the certificate, run the following command on the server:

Certutil -Store My 0 ServerCert.cer

Next we need to copy the ServerCert.cer to the client machine and run the following command:

Certutil -Verify -Urlfetch ServerCert.cer

Please note that we will need to validate the output in the same way as we did on the server. Typically we will fail either because the chain doesn't validate or we can't access the CRL. The private key will not be relevant.

Note: Certutil.exe is included in the base OS for Windows Server 2003, Windows Vista, & Windows Server 2008. For Windows XP & Windows 2000 clients, we will need to make a directory and copy the following files from a Windows Server 2003 machine:

%Windir%\System32\Certutil.exe
%Windir%\System32\Certadm.dll
%Windir%\System32\Certcli.dll

ADDITIONAL CONSIDERATION

Subject Alternate Name (SAN)

A Subject Alternative Name (SAN) lets you connect to a domain controller or ADAM instance by using a name other than the computer's FQDN. The format would be as follows:

SAN="dns=MemberServer.Contoso.com&dns=ldap.contoso.com"

There is one caveat. The first name in the SAN has to match the FQDN of the server. The second name doesn't matter.

LDAP over SSL Ports

By default all LDAP over SSL connections to a domain controller go over port 636. This is hardcoded and cannot be changed.

However, for ADAM we specify the port during installation. To verify which port the ADAM instance is using, we can run the following commands:

C:\WINDOWS\ADAM>Dsdbutil
Dsdbutil: list instances

Instance Name: PKI
Long Name: PKI
LDAP Port: 389
SSL Port: 636
Install folder: C:\WINDOWS\
Database file: C:\Program Files\Microsoft ADAM\PKI\data\adamntds.dit
Log folder: C:\Program Files\Microsoft ADAM\PKI\data
Service state: Running

Dsdbutil:

To change the ports, we can modify the following registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ADAM_instance1\Parameters\Port SSL

FINAL THOUGHTS

Once all errors in the validation process have been resolved on both the client and the server, we should be able to make our LDAP over SSL connections.

To test the connection we recommend using LDP.exe which is part of the Windows Support Tools. First try to make a connection on the server itself. If this fails, then verify the server certificate requirements using the earlier documented steps.

If this succeeds then we know that the server is configured properly. We can then try running LDP on the client to see if we can make the connection. Again, if it fails, then go through the client certificate requirements.

At this point, if we still can't connect, then most likely the problem is not in the certificate. The problem may be that port

We can get a network trace to verify the SSL handshake

Using these steps should resolve the majority of the LDAP over SSL connectivity issues.

ADDITIONAL RESOURCES

938703 How to troubleshoot LDAP over SSL connection problems
https://support.microsoft.com/default.aspx?scid=kb;EN-US;938703

932834 You may be unable to connect to a Windows Server 2003-based domain controller by using LDAP over an SSL connection
https://support.microsoft.com/default.aspx?scid=kb;EN-US;932834

931351 How to add a Subject Alternative Name to a secure LDAP certificate
https://support.microsoft.com/default.aspx?scid=kb;EN-US;931351

911647 Applications that use Lightweight Directory Access Protocol (LDAP) over Secure Sockets Layer (SSL) do not perform revocation checking of the server certificate in Windows Server 2003 with Service Pack 1
https://support.microsoft.com/default.aspx?scid=kb;EN-US;911647

321051 How to enable LDAP over SSL with a third-party certification authority
https://support.microsoft.com/default.aspx?scid=kb;EN-US;321051

Appendix A: Configuring LDAP over SSL Requirements for AD LDS
https://technet2.microsoft.com/windowsserver2008/en/library/2f5d0612-75f9-4883-bf52-d11c7cda907f1033.mspx?mfr=true

Comments