Ghost Management Point: Duplicate MP entries in the Location Services Log

Hello Y ’all,

Here an interesting issue I recently work with, this is the case of the Ghost Management Point. Spooky? Ow Yea.

The Symptoms:

Client machines suddenly stopped getting new software, new updates, new policies.
Upon further investigation, LocationServices.log at the client machines, gave us the first clue:

LocationServices 14508 (0x38ac) Refreshing security settings over AD
LocationServices 14508 (0x38ac) Performing AD query: '(&(ObjectCategory=mSSMSManagementPoint)(mSSMSDefaultMP=TRUE)(mSSMSSiteCode=PRI))'
LocationServices 14508 (0x38ac) OperationalXml '<clientoperationalsettings><version>5.00.8540.1005</version><securityconfiguration><securitymodemask>0</securitymodemask><securitymodemaskex>224</securitymodemaskex><httpport>80</httpport><httpsport>443</httpsport><certificatestorename></certificatestorename><certificateissuers></certificateissuers><certificateselectioncriteria></certificateselectioncriteria><certificateselectfirstflag>1</certificateselectfirstflag><pkicertoptions>0</pkicertoptions><sitesigningcert></sitesigningcert></securityconfiguration><rootsitecode>PRI</rootsitecode><ccm> <commandline>SMSSITECODE=PRI</commandline> </ccm><fsp> <fspserver></fspserver> </fsp><capabilities schemaversion="1.0"><property name="SSLState" value="0"></property></capabilities><domain value="contoso.local"></domain><forest value="contoso.local"></forest><aadconfig version="1.0"><timestamp>2017-11-03 10:37:03</timestamp><tenants></tenants></aadconfig></clientoperationalsettings>'
LocationServices 14508 (0x38ac) !aadInfo.sTenantId.empty() &amp;&amp; !aadInfo.sClientAppId.empty() &amp;&amp; !aadInfo.sResourceUri.empty(), HRESULT=87d00215 (..\CcmToken.cpp,2337)
LocationServices 14508 (0x38ac) The MP name retrieved is 'MP01.contoso.local' with version '8540' and capabilities '<capabilities schemaversion="1.0"><property name="SSLState" value="0"></property></capabilities>'
LocationServices 14508 (0x38ac) MP 'MP01.contoso.local' is compatible
LocationServices 14508 (0x38ac) !aadInfo.sTenantId.empty() &amp;&amp; !aadInfo.sClientAppId.empty() &amp;&amp; !aadInfo.sResourceUri.empty(), HRESULT=87d00215 (..\CcmToken.cpp,2337)
LocationServices 14508 (0x38ac) The MP name retrieved is 'MP01.contoso.local' with version '8634' and capabilities '<capabilities schemaversion="1.0"><property name="SSL" version="1"></property><property name="SSLState" value="31"></property></capabilities>'
LocationServices 14508 (0x38ac) MP 'MP01.contoso.local' is not compatible
LocationServices 14508 (0x38ac) !aadInfo.sTenantId.empty() &amp;&amp; !aadInfo.sClientAppId.empty() &amp;&amp; !aadInfo.sResourceUri.empty(), HRESULT=87d00215 (..\CcmToken.cpp,2337)
LocationServices 14508 (0x38ac) The MP name retrieved is 'MP02.contoso.local' with version '8634' and capabilities '<capabilities schemaversion="1.0"><property name="SSL" version="1"></property><property name="SSLState" value="31"></property></capabilities>'
LocationServices 14508 (0x38ac) MP 'MP02.contoso.local' is not compatible
LocationServices 14508 (0x38ac) Retrieved 1 MP records from AD for site 'PRI'
LocationServices 14508 (0x38ac) Received client operational settings: <clientoperationalsettings><version>5.00.8540.1005</version><securityconfiguration><securitymodemask>0</securitymodemask><securitymodemaskex>224</securitymodemaskex><httpport>80</httpport><httpsport>443</httpsport><certificatestorename></certificatestorename><certificateissuers></certificateissuers><certificateselectioncriteria></certificateselectioncriteria><certificateselectfirstflag>1</certificateselectfirstflag><pkicertoptions>0</pkicertoptions><sitesigningcert></sitesigningcert></securityconfiguration><rootsitecode>PRI</rootsitecode><ccm> <commandline>SMSSITECODE=PRI</commandline> </ccm><fsp> <fspserver></fspserver> </fsp><capabilities schemaversion="1.0"><property name="SSLState" value="0"></property></capabilities><domain value="contoso.local"></domain><forest value="contoso.local"></forest><aadconfig version="1.0"><timestamp>2017-11-03 10:37:03</timestamp><tenants></tenants></aadconfig></clientoperationalsettings>.

Yes, I know what you are thinking, somewhere in one of the related domains we will find a duplicated msSMSManagementPoint for MP01, right? That is what I thought so too, but we could not find it, anywhere. Worse yet, more and more clients were failing to retrieve the proper Management Point, because of the duplicate Management Point that was set to be HTTP, the true MP01 and MP02 are in fact HTTPS configured.

The Investigation:

It is time to bring the big guns, yes, the Directory Services Team!
By the way, Thank you very much for the help Renato Pagan , Mike Resnick and Clint Koenig.
We initially through that there was some Domain Controller, some Domain that we were not thinking of that may contained the Ghost Management Point entry somewhere.

I initially found this article from the Scripting Guys,

Use PowerShell to Query Active Directory from the Console

And Did this PowerShell Monstrosity:

LocationServices 9172 (0x23d4) Performing AD query: '(&(ObjectCategory=mSSMSManagementPoint)(mSSMSDefaultMP=TRUE)(mSSMSSiteCode=PRI))'

 ((New-Object DirectoryServices.DirectorySearcher "ObjectCategory=mSSMSManagementPoint").findall()).Properties

The goal was to go through as many domains that were related to the client machine as possible.
Later on talking to Renato Pagan he suggested to use this PowerShell that could be used and also point the Domain Controller we directly contacting”

 Get-ADObject -SearchBase ("CN=System Management," + (Get-ADDomain).SystemsContainer) -LDAPFilter "(&(objectClass=mSSMSManagementPoint))" -Property * -Server DC01

And to find which Domain Controller the client machine was connected to, we did not use SET LOGONSERVER, in fact we used nltest, as this example:

 nltest /sc_query: contoso.local
Flags: 30 HAS_IP HAS_TIMESERV
Trusted DC Name \\DC01. contoso.local
Trusted DC Connection Status Status = 0 0x0 NERR_Success
The command completed successfully

With this in hand we tried to run the PowerShell again, but still, we were unable to find the Ghost Management Point.
We also used ldifde queries to have a broader search, nothing

ldifde -f contoso_local.ldf -d dc=contoso,dc=local -r "(objectclass=msSMSManagementPoint)" -l cn,msSMSVersion

Things were getting desperate, and more and more clients were getting assigned to the Ghost Management Point. So, Mike Resnik suggested to capture a trace from the client machine, that would help us identify the specific LDAP query results we were getting from those clients. Two scripts:

START.CMD and STOP.CMD

START.CMD contains the capture, that includes the ccmxec.exe that is pushing the LDAP query, a network trace, and also enabling Verbose Logs for NetLogon for good mPRIure:

Reg add HKLM\System\CurrentControlSet\Services\ldap\Tracing\ccmexec.exe /f netsh trace start capture=yes maxsize=2048 tracefile=c:\temp\%computername%.etl REM Option to specify process ID (see KB 2221529): REM Reg add HKLM\System\CurrentControlSet\Services\ldap\Tracing\&ProcessName.exe& /v PID /t REG_DWORD /d PID /f REM You can also trace multiple processes at the same time by creating multiple registry keys logman create trace "ds_ds" -ow -o c:\temp\ds_ds.etl -p "Microsoft-Windows-LDAP-Client" 0x1a59afa3 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 -ets logman update trace "ds_ds" -p "Microsoft-Windows-ADSI" 0xffffffffffffffff 0xff -ets nltest /dbflag:ffffffff

STOP.CMD simply condensates the results, and saves them nice and pretty:

logman stop "ds_ds" -ets netsh stop Reg delete HKLM\System\CurrentControlSet\Services\ldap\Tracing\ccmexec.exe /f nltest /dbflag:2080ffff copy c:\windows\debug\netlogon.log C:\temp\.

So, we prepared the machine, by removing the registry value under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CCM\Security
And wiped the value of the key SigningCertificate to force the client to register.
From a command prompt as Admin we executed the START.CMD and started the ccmexec service.
Observing the LocationServices.log we saw the behavior happening and executed the STOP.CMD.

With the results in hand, it is time to look at them. So close now, I promise!
Using Microsoft Message Analyzer, we opened the ds_ds.etl from the results, and started looking for the 8540 version, that was being returned for the client, a quick tip is select all the message in Message Analyzer and paste in Notepad++, much faster to find what you are looking for, especially with text.
And, behold! There is your Ghost Management Point, first hand:

zCN=SMS-MP-PRI-MP01.CONTOSO.LOCAL\0ACNF:dee8d902-4b84-4eac-ab11-765116270f53,CN=System.Management,CN=System,DC=contoso,DC=local0.....0.......distinguishedName1....|.zCN=SMS-MP-PRI-MP01.CONTOSO.LOCAL\0ACNF:dee8d902-4b84-4eac-ab11-765116270f53,CN=System.Management,CN=System,DC=contoso,DC=local0....(..dNSHostName1.......MP01.contoso.local0....(..mSSMSMPName1.......MP01.CONTOSO.LOCAL0.......mSSMSVersion1.......85400.......mSSMSCapabilities1....w.....q5.00.8540.1005

The light at the end of the tunnel:

Great, we found it, now what?
So, this, this is a conflict record in active directory, that happened somehow during a Domain Controller replication or other situation in which two records with the same name existed simultaneously.
To fix that? LOL, yes, I am serious, LOL…

Description of the Lingering Object Liquidator tool Lingering Object Liquidator (LoL)

Once we used the Lingering Object Liquidator, LoL, and Purged the Unclean, ok, Dawn of War references at this point, seriously Renato? The good news is, once the Object was removed from Active Directory the clients are now able to properly return the values for the two existing Management Points, register, and live happily ever after.