It turns out that weird things can happen when you mix Windows Server 2003 and Windows Server 2012 R2 domain controllers
UPDATE: The hotfix is now available for this issue! Get it at https://support.microsoft.com/kb/2989971
This hotfix applies to Windows Server 2012 R2 domain controllers and should prevent the specific problem discussed below from occurring.
It’s important to note that the symptoms of users and computers not being able to log on can happen for a number of different reasons. Many of the folks in the comments have posted that they have these sorts of issues but don’t have Windows Server 2003 domain controllers, for example. If you’re still having problems after you have applied the hotfix, please call in a support case so that we can help you get those fixed!
=====================================================
We have been getting quite a few calls lately where Kerberos authentication fails intermittently and users are unable to log on. By itself, that’s a type of call that we’re used to and we help our customers with all the time. Most experienced AD admins know that this can happen because of broken AD replication, unreachable DCs on your network, or a variety of other environmental issues that all of you likely work hard to avoid as much as possible - because let’s face it, the last thing any admin wants is to have users unable to log in – especially intermittently.
Anyway, we’ve been getting more calls than normal about this lately, and that led us to take a closer look at what was going on. What we found is that there’s a problem that can manifest when you have Windows Server 2003 and Windows Server 2012 R2 domain controllers serving the same domain. Since many of you are trying very hard to get rid of your last Windows Server 2003 domain controllers, you might be running into this. In the case of the customers that called us, the login issues were actually preventing them from being able to complete their migration to Windows Server 2012 R2.
We want all of our customers to be running their Active Directory on the latest supported OS version, which is frankly a lot more scalable, robust, and powerful than Windows Server 2003. We realize that upgrading an enterprise environment is not easy, and much less so when your users start to have problem during your upgrade. So we’re just going to come out and say it right up front:
We are working on a hotfix for this issue, but it’s going to take us some time to get it out to you. In the meantime, here are some details about the problem and what you can do right now.
Symptoms include:
1. When any domain user tries to log on to their computer, the logon may fail with “unknown username or bad password”. Only local logons are successful.
If you look in the system event log, you may notice Kerberos event IDs 4 that look like this:
Event ID: 4
Source: Kerberos
Type: Error
"The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/myserver.domain.com. This indicates that the password used to encrypt the Kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (domain.com), and the client realm. Please contact your system administrator."
2. Operating Systems on which the issue has been seen: Windows 7, WS2008 R2, WS2012 R2
3. This can affect Clients and Servers(including Domain Controllers)
4. This problem specifically occurs after the affected machine has changed its password. It can vary from a few minutes to a few hours post the change before the symptoms manifest.
So, if you suspect you have a machine with this issue, check when it last changed its password and whether this was around the time when the issue started.
This can be done using repadmin /showobjmeta command.
Example:
Repadmin /showobjmeta * “CN=mem01,OU=Workstations,,DC=contoso,DC=com”
This command will get the object metadata for mem01 server from all DC’s.
In the output check the pwdlastSet attribute and see if the timestamp is around the time you started to see the problem on this machine.
Example:
Why this happens:
The Kerberos client depends on a “salt” from the KDC in order to create the AES keys on the client side. These AES keys are used to hash the password that the user enters on the client, and protect it in transit over the wire so that it can’t be intercepted and decrypted. The “salt” refers to information that is fed into the algorithm used to generate the keys, so that the KDC is able to verify the password hash and issue tickets to the user.
When a Windows 2012 R2 DC is promoted in an environment where Windows 2003 DCs are present, there is a mismatch in the encryption types that are supported on the KDCs and used for salting. Windows Server 2003 DCs do not support AES and Windows Server 2012 R2 DCs don’t support DES for salting.
You might be wondering why these encryption types matter. As computer hardware gets more powerful, older encryption methods become easier and easier to break. Thus, we are constantly incorporating newer, more powerful encryption into Windows and Kerberos in order to help protect your user passwords (and your data and your network).
Workaround:
If users are having the problem:
Restart the computer that is experiencing the issue. This recreates the AES key as the client machine or member server reaches out to the KDC for Salt. Usually, this will fix the issue temporarily. (at least until the next password change).
To prevent this from happening, please apply the hotfix to all Windows Server 2012 R2 domain controllers in the environment.
How to prevent this from happening:
Option 1: Query against Active Directory the list of computers which are about to change their machine account password and proactively reset their password against a Windows Server 2012 R2 DC and follow that by a reboot.
There’s an advantage to doing it this way: since you are not disabling any encryption type and keeping things set at the default, you shouldn’t run into any other authentication related issue as long as the machine account password is reset successfully.
Unfortunately, doing this will mean a reboot of machines that are about to change their passwords, so plan on doing this during non-business hours when you can safely reboot workstations.
We’ve created a quick PowerShell script that you can run to do this.
Sample PS script:
> Import-module ActiveDirectory
> Get-adcomputer -filter * -properties PasswordLastSet | export-csv machines.csv
This will get you the list of machines and the dates they last set their password. By default machines will reset their password every 30 days. Open the created csv file in excel and identify the machines that last set their password 28 or 29 days prior (If you see a lot of machines that have dates well beyond the 30 days, it is likely these machines are no longer active).
Reset Password:
Once you have identified the machines that are most likely to hit the issue in the next couple of days, proactively reset their password by running the below command on those machines. You can use tools such as psexec, system center or other utilities that allow you to remotely execute the command instead of logging in interactively to each machine.
nltest /SC_CHANGE_PWD:<DomainName> /SERVER:<Target Machine>
Then reboot.
Option 2: Disable machine password change or increase duration to 120 days.
You should not run into this issue at all if password change is disabled. Normally we don’t recommend doing this since machine account passwords are a core part of your network security and should be changed regularly. However because it’s an easy workaround, the best mitigation right now is to set it to 120 days. That way you buy time while you wait for the hotfix.
If you go with this approach, make sure you set your machine account password duration back to normal after you’ve applied the hotfix that we’re working on.
Here’s the relevant Group Policy settings to use for this option:
Computer ConfigurationWindows SettingsSecurity SettingsLocal PolicesSecurity Options
Domain Member: Maximum machine account password age:
Domain Member: Disable machine account password changes:
Option 3: Disable AES in the environment by modifying Supported Encryption Types for Kerberos using Group Policy. This tells your domain controllers to use RC4-HMAC as the encryption algorithm, which is supported in both Windows Server 2003 and Windows Server 2012 and Windows Server 2012 R2.
You may have heard that we had a security advisory recently to disable RC4 in TLS. Such attacks don’t apply to Kerberos authentication, but there is ongoing research in RC4 which is why new features such as Protected Users do not support RC4. Deploying this option on a domain computer will make it impossible for Protected Users to sign on, so be sure to remove the Group Policy once the Windows Server 2003 DCs are retired.
The advantage to doing this is that once the policy is applied consistently, you don’t need to chase individual workstations.However, you’ll still have to reset machine account passwords and reboot computers to make sure they have new RC4-HMAC keys stored in Active Directory.
You should also make sure that the hotfix https://support.microsoft.com/kb/2768494 is in place on all of your Windows 7 clients and Windows Server 2008 R2 member servers, otherwise they may have other issues.
Remember if you take this option, then after the hotfix for this particular issue is released and applied on Windows Server 2012 R2 KDCs, you will need to modify it again in order to re-enable AES in the domain. The policy needs to be changed again and all the machines will require reboot.
Here are the relevant group policy settings for this option:
Computer ConfigurationWindows SettingsSecurity SettingsLocal PolicesSecurity Options
Network Security: Configure encryption types allowed for Kerberos:
Be sure to check: RC4_HMAC_MD5
If you have unix/linux clients that use keytab files that were configured with DES enable: DES_CBC_CRC, DES_CBC_MD5
Make sure that AES128_HMAC_SHA1, and AES256_HMAC_SH1 are NOT Checked
Finally, if you are experiencing this issue please revisit this blog regularly for updates on the fix.
- The Directory Services Team
Comments
- Anonymous
July 23, 2014
Hey y’all, Mark here again with a quick announcement. Our brothers in arms at AskDS have just posted - Anonymous
July 23, 2014
thank you - Anonymous
July 24, 2014
Hi David,
Thanks for posting this. We are currently affected by this after a 2003 to 2012R2 AD upgrade. Note the article title suggests this happens only when 2003/2012R2 DCs are mixed, however the problem continues even AFTER the 2003 DCs have been removed from the domain. We are now only running 2012R2 DCs on 2012R2 FL, and the problem remains.
Its also worth highlighting that when a DC itself is affected by the issue, it stops sharing SYSVOL and clients can no longer talk to it, nor can you log onto the DC with any account. If you have 2 DCs and they reset at the same time say over a weekend, no one in the domain can login on Monday morning. We are also seeing machines happening to reset their password whilst the user has left it on a locked session. They are then unable to log back in when they return, and are unfortunately losing work. I'm sure you are aware of these additional observations, but just thought it worth mentioning in case not
Hoping for a fix very soon!
Mark - Anonymous
July 24, 2014
David - appreciate the post. Had this exact issue about a month ago and MSFT had me increase secure channel expiration as you said. Glad to see it wasn't just, and Microsoft has recognized the issue and is working on a fix. - Anonymous
July 24, 2014
What about 2012 R1 (one) DCs? Is it just R2 that does not use DES for the salt? - Anonymous
July 24, 2014
The comment has been removed - Anonymous
July 24, 2014
Also, when trying to run nltest against the CNO I just get access denied. - Anonymous
July 24, 2014
I believe that Premier Field Engineering has a Best Practices Analyzer that they've developed for AD. It seems like it would be a really good idea to have DcPromo run this as its first step. And as my colleague Brian points out, why doesn't AD negotiate a mutual cipher suite between client and DC in this case? Either don't allow incompatibilities to happen, or test and warn before creating such a situation. - Anonymous
July 24, 2014
Similar issue was there mix environment of 2003 and 2008 R2 DCs when 2003 server holds FSMO roles. KDC Event ID 27, I think both the issues looks similar to me. If they are different please explain how?
http://support.microsoft.com/kb/977321 - Anonymous
July 24, 2014
I have a current ticket open with Microsoft right now. Opened it on 7/19. My TAM sent me this post this morning. We decommissioned our last 2003 R2 domain controller in June 2014. We introduced 2012 R2 domain controllers in April 2014. We've had 2008 R2 domain controllers for years. We have 2008 R2 and 2012 R2 member servers that we cannot login after the password reset. We have to logon locally and reboot the server. We have about 5 servers per day in a 1800 server environment. It's getting pretty painful. We were wondering if we raised our Forest Functional Level to 2008 R2 that maybe the issue would go away. Hasn't happened with 2003/R2 member servers or our Win7 workstations. But our standard troubleshooting at the service desk could be masking workstation issues. We have reproduced the issue in one of our lab environments. The issue is getting painful problem is starting to who up on our 2012 R2 Hyer-V hosts. - Anonymous
July 24, 2014
Any chance this could cause similar issues on a 2008 RTM / 2012 R2 combo if the Domain Functional Level is 2003? We elevated to 2008 after reading this post because we were having some bizarre uncharacteristically long logon times from our servers after bringing in a 2012 R2 DC into the mix. - Anonymous
July 24, 2014
Would this effect a trust between a 2012 r2 domain and 2003, or 2008 r2 for that matter? - Anonymous
July 24, 2014
Hey guys,
Rob Greene from Directory Services. Let see if I can answer a few of these questions for the group:
This problem is very specific - You have to have Windows Server 2003 domain controllers in the domain, and then start to move to Windows Server 2012 R2 domain controllers to replace the 2003 domain controllers.
Windows Server 2012 domain controllers are not affected by this DES not supported so therefore no SALT created.
We have not heard of any issues currently with Cluster CNO's. I know that Windows Server 2008 - 2008 R2 clusters CNO's actually do not support AES encryption and only support RC4_HMAC_MD5. Not sure actually for Windows Server 2012 Clusters if that is what you are running.
As far as your comment about running NLTest against the CNO what exactly are you trying to do with the CNO object using NLTest?
As far as needing to do something in DCPromo this is not a DCPromo problem this is a Kerberos authentication. It is being worked by the development team as a bug and is going to be fixed. Please just bear with us while this process happens they need time to thoughtfully plan the fix, and then do testing to verify it is not going to cause more issues.
The Kerberos client and the KDC are actually negotiating the authentication. The KDC is searching and using the strongest encryption type supported by itself and the end service which AES is the strongest encryption on 2008 and higher computers.
For those of you who have decommissioned your last Windows Server 2003 domain controller after adding Windows Server 2012 R2 DC's, all that you should need to do is force the password change one time with the 2012 R2 domain controllers. The issue can actually take up to two password changes to happen and clear up from talking with our Escalation Engineers.
This problem is not going to cause a slow logon problem. This issue causes a NO LOGON issue. If you are experiencing a slow logon problem you might want to determine if it is an issue with connectivity to the domain, group policy issues, logon script issues.
In theory this should not affect trusts as long as the trusts are not configured to use AES encryption. By Default the should be using RC4_HMAC_MD5. In Windows Server 2008 and higher you can modify the trust to support AES encryption however I do not believe that Windows Server 2003 will know how to handle that trust password (Inter-realm ticket). - Anonymous
July 25, 2014
Thanks for this information! - Anonymous
July 25, 2014
Hi Rob, Thats very useful information thankyou for posting. Unfortunately Support are giving much conflicting information over this. We are running 2012R2 only, and only this week I have been advised we will continue seeing the issue every 30 days, and that we still need to implement the workarounds. Typically I implemented option 2 only last night, before reading your post this morning! - Anonymous
July 25, 2014
We are facing similar issues, coming from a 2003 domain, adding 2012R2 DC's.
I saw Kerberos authentication failing because the newer clients use AES encryption type for Kerberos and the domain was still running 2003 DFL (supporting DES and RC4) so encryption type not supported was logged and RC4 was used. My assumption was rasing the DFL to 2008 would fix this issue because AES enctype is supported.
We have fileover clusters running 2012, RDS servers running 2008R2 and DC's running 2012R2.
If the fileshare resource runs on one node we can successfully browse and access the fileshare, when we failover the resource we get access denied. what is strange is the Security-Kerberos ID4 error KRB_AP_ERR_MODIFIED from the server "nodename" The target named used was cifs/filesharename.domain.local.
Yes we have correct spn's, even tried rejoining domain, repair cluster/fileserver resource and the same spn's get registered. Same compared to other similar configured environments were we didn't see these errors so far.
Should we open a support-case to receive this hotfix or will it be public available?
regards,
Matthijs Wijers - Anonymous
July 25, 2014
I can confirm enabling policy "Domain Member: Disable machine account password changes", resetting the computer account password (and rebooting the server) works. Kerberos authentication works again with AES encryption now, can browse to the failover cluster fileshare resource running on each node now.
regards,
Matthijs Wijers - Anonymous
July 25, 2014
The comment has been removed - Anonymous
July 25, 2014
Matthjis, eventually the hotfix will be publicly available. Right now we're still working on building it so obviously we don't have it for anyone yet :) Our best guess right now is that we're probably a few weeks from being able to post a public KB with the hotfix, but a lot depends on how testing goes.
We'd have to take a look at your environment to be able to really explain why you're seeing ticket requests with that target principal name in them fail. It may be due to the way you set up your file server resources in your cluster. You can also see that happen if you're using a domain-based DFS root. Or, it could be a particular application that's just asking for the wrong resource entirely - remember that the KDC attempts to issue tickets based on what the calling client or application asks for. If an application asks for something that's not correct (but that exists), the KDC will attempt to honor that request. It's just impossible to say without knowing a lot more about your implementation, so a support case is definitely the best option for that particular issue. - Anonymous
July 25, 2014
Oops, missed your post on the second page here! Awesome that you got it working! Don't forget once the hotfix releases to re-enable the policy for machine account password changes.
David - Anonymous
July 25, 2014
The comment has been removed - Anonymous
July 25, 2014
The comment has been removed - Anonymous
July 26, 2014
While working a support case with MS regarding this issue, we were informed that best practice when upgrading a domain was to remove ALL computers from the domain and rejoin them to the domain. I can see how this would have resolved the issue, but didn't appear in any of the documentation that I have read. - Anonymous
July 27, 2014
We are just about ready to begin our 2003 to 2012 R2 migration - so glad I found this post! Is MS confident in the workarounds or is the best course of action to wait for the hotfix? Is upgrading AD to 2008 (or 2012?), then to 2012R2 an acceptable option? Thanks! - Anonymous
July 28, 2014
Discovered this issue in our AD today. Very critical!
Why is it the PC can renew its password on the fly, without rebooting the entire system, and when we force it to change its password, we have to reboot?
Is there no way to trigger this without rebooting?
Also when we set the Maximum MachineAccount password age, do we have to reboot to make it pick up this setting also?
Would like to avoid rebooting as much as possible. - Anonymous
July 28, 2014
I have yet to upgrade from 2003 - at this point can I simply upgrade to 2008 R2 first to avoid these issues? And should I stay on 2008 R2 DC for any period of time, or simply proceed to 2012 R2 the same day? - Anonymous
July 28, 2014
Same problem here, we have this issue for 6 - 8 weeks now. We started to migrate our domain from Windows Server 2003 R2 to Windows Server 2012 R2 and randomly one of our clients / servers (Win7, 2008 R2 and newer) logged the event log Event ID: 4 / KRB_AP_ERR_MODIFIED and our users were not able to login with domain account. So far, our new DC (2012 R2) were not affected by the issue till now.
We opened a ticket at Microsoft support 10 days ago, the support was analyzing our logs and told us to install KB2768494 what we did on Thursday (24.07.2014) in the first step for our terminal and hyper-v host servers only. But one day later a terminal server was affected by the issue and KB2768494 was already installed. Today (Monday 28.07.2014) in the morning couple of servers (most of them terminal servers) had the login issue as well and our user were not able to log in. A restart of the affected servers solved the problem temporary...
We found this blog and tried to get used to option1, so we send the command (NLTEST /SC_CHANGE_PWD) to each server to renew his Kerberos ticket with AES, and only view minutes later couple of servers went into this strange behavior immediately, one of them is a hyper-v host server. :-(
General environment information: On the last weekend we demote every of our Windows Server 2003 DC and replaced them with Windows Server 2012 R2 as DC. We raised our domain level to 2008 including SYSVOL_DFSR. What else we can do? Option2 and Option3 need a reboot of each server again which is not handy in our environment. Is it normal that couple of servers went into this Kerberos issue right after "NLTEST /SC_CHANGE_PWD" or is it just "random"?
Robert - Anonymous
July 28, 2014
We had this issue as well once we upgraded all our domain controllers from 2003R2 to 2012 R2 including the domain functional level. I know all the servers had the issue at least once. It is interesting to see now that they will have it twice. I am not sure if I want to allow that to happen or if I want to keep the disable the Machine password resets until the fix has been released.
I will also mention we only had the issue with servers and not with workstations (most are win 7). - Anonymous
July 28, 2014
Hey Guys Rob Greene again,
lets see if I can answer some more of these comments:
Thanks The Duder, and Lesta G for the comments.
BurntAdmin - I am sorry that you are getting conflicting information. However in general there should never be a reason to remove all the computers from the domain before / after upgrading your domain to a new version of Windows. I am not aware of any myself, but I guess there might be certain situations that might require you to remove a computer from the domain (Windows NT Computers come to mind only) before you upgrade.
arguea - Again thanks for the comment. I am glad that our readers and customers understand that we are trying to do the right thing and let you know of this problem since everyone is in the mad dash to get rid of Windows Server 2003 before end of support.
Now as far as your question, you most definitely could upgrade to Windows Server 2008/R2 domain controllers... Heck even Windows Server 2012 and avoid this problem. However you would need to wait until all the computers have gone through two password changes before installing Windows Server 2012 R2 domain controllers otherwise you will still be in the same boat.
AdminGr / Mobay - Sorry Mobay, none of the workaround are going to prevent you from needing to reboot the computers for them to take affect.
Auraria - Please look at my comment for Arguea... You both have basically the same question.
Bernard Welmers - be ready for the workstations to have some issues, unless they are not using AES encryption at all. You might want to look and see if somehow you implemented the supported encryption types on the workstations possibly. Or verify that your workstations are actually changing their password every 30 days (Default). - Anonymous
July 28, 2014
Thanks for the answers. greener85
Why is restart requirement set to none on this Technet page, if we still need to restart?
http://technet.microsoft.com/en-us/library/jj852252(v=ws.10).aspx
Also. What about CNO (cluster computer accounts) Are they affected by this issue? Do they even update theire passwords and are they able to apply the GPO if so? - Anonymous
July 28, 2014
So, does KB 948963 not help with this? That contains a hotfix for Server 2003 that adds the AES cipher suites to SCHANNEL.
An update is available to add support for the TLS_RSA_WITH_AES_128_CBC_SHA AES128-SHA and TLS_RSA_WITH_AES_256_CBC_SHA AES256-SHA AES cipher suites in Windows Server 2003
http://support.microsoft.com/kb/948963
I do not have easy access to any Server 2003 instances these days to look in to this with. - Anonymous
July 29, 2014
Hi Rob,
In our all 2012R2 Domain, lets assume we go for Option 4) Ride it out - knowing we only have to get past 2 password resets, can you tell us if there is a way to query a machine to determine if it has completed it's 2 password resets and is now considered a fixed machine? Obviously it would be immensely useful if there is a way we could determine this and then script it. Desktops aren't so much of a problem, as most of these get rebooted daily. Its the hundred's of servers we need to determine the status of. This would be a great help. Can you also confirm that machines built/rebuilt AFTER we completed our upgrade (and removed the W2K3 DCs) will not be affected by this issue. My understanding is they will pickup the correct Kerberos tickets from day 1 and be unaffected, but the support desk stated they would still be affected. TIA. Mark - Anonymous
July 30, 2014
Thanks for this post, however you should make it easier to find ;) We were facing this issue for a couple of weeks now and tried a lot of things without effect. Finally on the point were we went out of options, I tried a last search with different key words and luckily found this post. :D
I'm a little confused about how many times a password needs to be changed. You mention that if a user is having this problem you have to reboot, but you still have to change the machine account password again to prevent it from happening again after 30 days.
However if you go for option 1 and proactively change the password I understand you only have to change it once. What is the difference between a machine changing the password itself after 30 days and forcing a change with nltest. To my understanding they both create a new AES key or am I missing something?
Another thing we noticed is that after the password is changed the local administrator account password is expired and set on "User must change password at next logon". Can this be prevented or is this expected behavior?
Again thanks for the post and waiting your answers and hopefully soon the Hotfix!
Dennis - Anonymous
July 30, 2014
Thank you for answering the questions in the comments. They have helped me determine that we should not be affected by this. We moved off of 2003 DCs a couple years ago to 2008 R2 and recently upgraded them all to 2012. I am confident with your answers that if we decided to upgrade them all to 2012 R2 we would not experience this issue (since the time elapsed from removing all 2003 DCs has been so long) - Anonymous
July 30, 2014
The comment has been removed - Anonymous
July 30, 2014
The comment has been removed - Anonymous
July 30, 2014
The comment has been removed - Anonymous
July 30, 2014
The comment has been removed - Anonymous
July 31, 2014
Hi David and team,
thanks for clarifying things and answering the questions!
I've still a question about how to tell if a change happened or not.
I've noticed that if you give the command "klist -li 0x3e7" on a 2008R2 or 2012R2 server to list the machine account tickets, you will notice a difference on the tickets where Client and Server is equal to "$ @". (this is the actual ticket referred to in the event id 4 error)
When a password isn't changed yet you see "KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)".
After the password is changed you will see "KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96"
For me this indicates that the key is changed to AES at least once. I've ran this command on multiple systems and all gave the same result. (on Windows 7 systems I used psexec to run the command remotely)
Is this correct to assume and use this as an indicator?
and can you still answer my question about the local administrator account password that is set on expired?
again thanks,
Dennis - Anonymous
July 31, 2014
some text parts were seen as code signs :s
the part " "$ @". " should be " HOSTNAME$ @ REALM " - Anonymous
July 31, 2014
Hey Dennis,
Great callout on using KList. I'm embarrassed, I actually forgot that it grabbed the EType on the issued ticket. But that's a good way to verify that you were able to get a ticket that used AES.
On the administrator account question - there's nothing in a machine account password change that should force the local administrator account password to expire. Are you using anything to manage the local administrator account password (a script or anything) or is the local administrator account named the same thing as the machine, perhaps? - Anonymous
August 04, 2014
218 Microsoft Team blogs searched, 47 blogs have new articles. 96 new articles found searching from 21 - Anonymous
August 04, 2014
The comment has been removed - Anonymous
August 05, 2014
Same Problem here.
Two of our Three 2012 R2 Hyper-V Cluster Nodes "stopped" working because of this bug.
No connection to the Cluster is possible right now.
Luckily the Guest Machines keep on running....
Lets hope I don`t get a split-brain on the cluster.
Support Ticket is opend. - Anonymous
August 05, 2014
Just wanted to say that I am having the same exact issues as "Markey165" is having. We had Server 2003dcs and are still having this issue after we have migrated to Server 2012R2 and Server 2008R2 DCs. We no longer have any 2003 DCs yet still have the same issue. - Anonymous
August 06, 2014
Hi all,
As we would like to upgrade our 2003 domain we followed a certain migration path. Running on AD 2003 DFL with 2003 R2 DC's, added 2 DC's 2012R2 then we changed the priority (priority, weight change) so our 2003 DC's became second and all requests were handled by the new 2012 DC's. At that moment the problems decribed in this blog started. After the promote we discovered the replica partners were not found immediately on the other DC's and we have to force them, this was weird. We demoted the servers promoted them again, same result. We thought our problem was caused by bad replication we made sure the repica partners were visible on all dc's and left the situation as is, but situation went worse. Our citrix PVS resets computer account passwords each 7 days, so each day we had several citrix servers that did not accept user credentials anymore. Strange enough the hit ratio this happened was random, sometimes only 1 servers sometimes 5 or more.
At this time we found out the computer account password was the problem and I made an excel extracting all computer objects with the pwdlastset attribute, so we could "predict" the next computer to fall out.
We also discovered that you can reset the account with powershell but the powershell has to be V3
logon to the server as local admin
You can test with: test-computersecurechannel
fix it with: Test-ComputerSecureChannel -Repair -Server (DC you want to use) -Credential (get-credential)
this works for us but important: if your powershell does not recognise "-credenial" it won't work and you'll have to upgrade your powershell
Anyway, we are facing this problem for 2 months and finally bumped into this blog saying it's a bug. So we decided to reverse the priority, the 2012 DC's are still in the domain but have less priority all request are hadled by the old DC's and so far, servers aren't facing the issue anymore.
Finding this post was not easy and finally landed on this blog thanks to the reply from someone on this blog.
Please make this blogpost more visible to users in some way because this issue makes and admin doubt about everything in his network :-) at least we did spent a lot of hours in troubleshooting this one.
Reagards
Wim - Anonymous
August 07, 2014
for those that have ditched their 2003dcs and still having issues, have you tried upping the domain and forrest functional levels to 2012R2 ? any difference? about do migrate this saturday and would be nice to know! - Anonymous
August 07, 2014
In reply to qrumdada2.
We are running 2008 R2 level now, and it does not make a difference. . I can recommend to increase duration of password change to 180 days. Hopefully we will have a hotfix before this expires. Again :-) - Anonymous
August 08, 2014
Unfortunately we're still seeing this issue even though all 2003 DCs have been demoted and removed from the domain. We started with two 2003 DCs and put in two 2012 R2 DCs in May. The 2003 DCs were demoted and removed by June and DFL/FFL was raised to 2012 R2.
We haven't opened a case with MS yet since it seems others are also still having this issue post-2003 DC removal. For the time being we have set computer account passwords to expire after 120 days. - Anonymous
August 09, 2014
@MRCUR It seems several of us are still seeing this after the 2003 DCs have been removed. We were also still seeing this after having raised the Domain and Forest FLs to 2012R2 (to answer qrumdada2's point).
David from the DS team replied to say that those of us seeing the problem with all 2012R2 DC's are seeing a different issue, and should log a call with support. However I think there is some confusion here, as I believe we are all still seeing this shortly after removing the 2003DCs, and Rob Greene's earlier reply said that after you remove the last 2003 DCs, you still need to get past at least 1 further password reset on the new 2012R2 DCs after which you should be in the clear.
I am assuming the latter statement from Rob is the correct position on this. Perhaps you could confirm this as a sanity check for us Rob/David? :) - Anonymous
August 11, 2014
I just got off the phone with Microsoft Software Assurance Support and I have the exact same issue as above.
We had 5 Domain Controllers before:
----------------------------------------
Forest/Domain Functional Level 2003
4x Server 2003 (one holding FSMO roles)
1x Server 2008R2
After migration to
-----------------------------
Forest/Domain Functional Level 2003
1x Server 2008R2
4x Server 2012R2 (one holding FSMO roles)
(no more server 2003 dcs but still having issue, ad some AD computers still have wrong hashes in AD)
We are still having the issue but some machines that have had 2 password resets do not seem to be having these issues anymore. I have been actively restarting some and changing the machine passwords while some I am going to wait for an update to fix this. - Anonymous
August 11, 2014
We were just preparing to perform a 2003 --> 2012 R2 domain upgrade. We will definitely be waiting now! Thanks for the good info in the blog!!! - Anonymous
August 12, 2014
Thank you for this, good to know and can be used when assisting clients. - Anonymous
August 15, 2014
Is there any word yet on when the hotfix will be ready? We are inching closer and closer to being ready for our upgrade and I would rather have a fix than use a workaround! - Anonymous
August 17, 2014
Hi DS team,
Any news in regards to this issue? When do you ship the hotfix?
/Peter - Anonymous
August 19, 2014
@Rob Greene
Anything new on the CNO from the Cluster Team?
We have a Server 2012 R2 Cluster running and had some mean issues with our 2012 R2 Domain Controllers after last "Update Tuesday"
Regards
Para el Mundo - Anonymous
August 19, 2014
Good morning,
the first days of september I should make a migration, are there any news about the hotfix? - Anonymous
August 20, 2014
The comment has been removed - Anonymous
August 22, 2014
Any update on this? We are ready to migrate to our 2012 R2 servers but because we came across this issue we are holding off until the fix is out. - Anonymous
August 22, 2014
Good thing this was found one year before Server 2003 exits extended support! - Anonymous
August 25, 2014
Here is the work-around that I have come up with from all of the feedback that I have found on this issue:
This PowerShell command will export all of your computers in Active Directory and when their passwords reset.
Get-adcomputer -filter * -properties name,PasswordLastSet | Select-Object name,passwordlastset | sort passwordlastset | Export-Csv c:passwordlastset.csv
Find all of your accounts that reset 28 days ago THAT ARE NOT CLUSTERNAMES OR NON-WINDOWS DEVICES and issue this command:
nltest /SC_CHANGE_PWD:DOMAIN /SERVER:workstation
If you issue this command before it stops communicating, you do not have to reboot. If it has already started failing on the authentication, a reboot will resolve the issue.
I usually wait for replication between DCs and run the PowerShell command again.
We previously upgraded DCs from 2003 to 2012 R2, but we have demoted all of our 2003 Domain Controllers. We are still having a problem on the day that the computers try to update their computer account passwords. It was pretty scary when all of our 2012 R2 DCs expired the same day because they were all joined to the domain on the same day. Luckily we had one that hadn't expired yet that we could log into. - Anonymous
August 25, 2014
Wondering if this (http://support.microsoft.com/kb/978055)couldn't be related as a distant cousin, since some have mentioned that it didn't work when they went back to Server 2008 R2. I have a separate environment where I did have a problem going from 2003 to 2012 R2. I demoted my 2012 R2 DC spun up a 2008 R2 DC and I am still having domain issues. - Anonymous
August 26, 2014
We are experiencing the exact same issue at two of our customers.
We have one customer environment where the mix of both Operating systems is still in play. The other customer environment is one where all windows 2003 servers DC's were demoted. In the latter environment the domain and forest functional level has been raised to 2012 R2, but nonetheless the issue still remains.
Is this issue still followed up by the Directory Services team or do we need to open a Premier support call for this? - Anonymous
August 28, 2014
Hello DS Team
Is this Hotfix already out there ? ;-) we still have this issue in some of our Domains (same Forest) and migrating all the DCs from one Domain to Server 2012 R2 is not solving this issue! just to notify you how urgent it is to get a fix. Any update would be very appreciated since it is already end of August now!.
regards
Ueli - Anonymous
August 28, 2014
Hi everyone,
Just a quick note that the hotfix is now available: http://support.microsoft.com/kb/2989971
I've added this to the main text of the KB article as well.
There have been several comments over the past week that we want to respond to, but haven't had a chance to get to yet. I want you all to know that we're not ignoring you and will try to keep responding to comments on this post for as long as necessary - it may just sometimes take us a few days to get back with a response. - Anonymous
August 28, 2014
The prerequisite information in the hotfix is unclear. Does KB2919355 need to be applied to ALL Windows 8 machines on the network prior to installing the hotfix or does it only need to be applied to the 2012 R2 servers? Or does KB2919355 need to be applied to both (Windows 8 and 2012 R2) prior to installing the hotfix? Easier to apply to a few servers than hundreds of Windows 8 machines so really need the clarification on this. Thanks - Anonymous
August 28, 2014
Both hotfixes need only be installed on domain controllers - not on clients, not on member servers. We'll see about making this more clear in the KB, thanks for the feedback. - Anonymous
August 29, 2014
We have completed our domain controller upgrades and no longer have any Windows 2003 domain controllers.
Would you still recommend this patch be applied since there was a time period of several weeks in which both Windows 2003 and Windows 2012 R2 domain controllers were running together? - Anonymous
August 29, 2014
If I apply the KB2989971 hotfix before promoting the Windows 2012 R2 member server to a domain controller do I then need to reapply the hotfix after it has been promoted to a DC? Or is this a one-time installation? - Anonymous
August 29, 2014
@djcgmcse - Yes, to be safe you should go ahead and apply the hotfix to your Windows Server 2012 R2 domain controllers. That will help insure that if you have any computer accounts out there that don't have AES credentials populated yet, the new DCs will still handle that correctly.
@DariceIT - You should only need to apply the hotfix one time. Once the binaries are updated on the servers, those will persist even if you install or uninstall the Active Directory role. - Anonymous
August 29, 2014
@Stan: I'd recommend trying the hotfix and if you still see problems in your customer environments definitely give us a call - that means there's something else going on that should be investigated. - Anonymous
August 29, 2014
@ Mike Whalen
So first, I'm sorry that you had such a bad experience with that prior support call. That's obviously not what we strive for here and I hope that you took the opportunity to fill out a survey on that ticket. We take those surveys very seriously, and they get reviewed by every level of management all the way up to our executives. You also always have the right to speak with a manager if you're working on a support issue with us and things aren't going well.. I hope that you'll be willing to give our support teams a second chance at some point, if you run into another issue.
Anyway, if you're running into problems with Kerberos authentication and you want to try to troubleshoot it yourself, I'd recommend starting with a network capture of the failing authentication. This will help you narrow down where the failure is occurring and allow you to isolate the account or servers involved. From there, it's a good idea to go through the event logs and see whether there's something failing that might impact Kerberos, like AD Replication, or whether there's another problem on the machine that might be contributing to the issue.
You can also get some free diagnostics from here to help in troubleshooting: https://home.diagnostics.support.microsoft.com/SelfHelp?wa=wsignin1.0
The Windows Directory Services Diagnostic listed on that page is the same one that we have our own engineers use to gather data and do automated analysis to look for common/known issues. - Anonymous
August 30, 2014
@ David Beach - You believe I should not apply this hotfix, then? - Anonymous
August 30, 2014
thanks - Anonymous
September 01, 2014
@ David Beach
nmitteam posted a question on 28 Aug regarding the prerequisite for this hotfix. Can you please answer his question or at least answer mine?
The hotfix states "To apply this hotfix, you must first install update 2919355 on Windows 8.1 or Windows RT 8.1." Why do we need that update 2919355 first? Does it need to be installed on the 2012 R2 Domain Controllers first or where should we install it? - Anonymous
September 02, 2014
Hey RhodanNZ,
The reason why you have to have the KB: 2919355 on the domain controller is because we no longer create Service Packs for operating systems. They now refer to a SP as an Update, so basically you can think of 2919355 as a Service Pack, and the hotfix has a service pack requirement of the April 2014 update.
So for the discussion of the hotfix the April 2014 update is only needed to be installed on the domain controllers.
However in general, you should apply the April 2014 Update on all your workstations and servers that it can be installed on.
Hopefully this makes some sense.
Rob Greene - Anonymous
September 02, 2014
Hi David/Rob,
I wonder if one of you could answer the question I posted 10 Aug 2014 3:15 AM to clarify the situation for several of us that have already removed Server 2003 DCs and were continuing to see the issue?
I believe we are merely seeing the tail end of the problem, and once affected machines are rebooted and get their AES encrypted password, they are fine from there on in.
If this is the case, the problem description provided is a little misleading, as it implies this only happens in a mixed environment, when actually it appears to continue in a pure 2012R2 environment until at least the next reset which could be another 30 days after the last 2003 DC is removed? At least that is my understanding.
It would be good to have specific confirmation on this point? - Anonymous
September 03, 2014
I have just tried to install the fix on a 2012 R2 Standard server and got a surprise. The Standalone Installer claims this update is not applicable to my machine. I checked for the prerequisite KB2919355 and it is installed. Any other ideas? - Anonymous
September 03, 2014
Hi Arguea,
I am able to reproduce that issue if the Active Directory Domain Services role is not installed. Run this Windows PowerShell command
Get-WindowsFeature domain
You should see:
[X] Active Directory Domain Services AD-Domain-Services Installed
If it is not installed, run:
Install-WindowsFeature ad-domain-services
Then your hotfix will install fine. - Anonymous
September 05, 2014
That worked! thank you! - Anonymous
September 08, 2014
Hi all,
I've been running the hotfix for 1 week now and haven't got any problems anymore. Looks like this hotfix fixed our problem. Thanks. - Anonymous
September 09, 2014
Hi guys,
I'd like to fix a Windows 2012 R2 STD but I don't want install the Active Domain Services. The server is used to deploy SEP.
How to fix it?
Best Regards. - Anonymous
September 09, 2014
If ADDS is not installed, there is nothing to fix - this hotfix updates the KDC Service, which you do not have installed. - Anonymous
September 09, 2014
I applied the hotfix to all six of our 2012 R2 DCs yesterday afternoon. I had previously implemented the 120 day computer account password change Group Policy as recommended in this post. After the hotfix was installed on all six DCs, I removed this policy (going back to the default 30 day password change).
Immediately after this, I could no longer log into one of my DCs. The Event ID 4 showed up in the Event Viewer after I rebooted the DC and was able to log in again. I manually changed the computer account password through PowerShell and implemented the 120 day policy once again for all domain computers.
Anyone else seeing this issue still occur after the hotfix? As I previously posted, we have NO 2003 DCs in the domain anymore. The only DCs in the domain are running 2012 R2, and the FFL/DFL is 2012 R2 as well.
Thanks! - Anonymous
September 09, 2014
We installed this patch the day it was released and have had two 2012 member servers experience the issue since then requiring logging in with a local admin account and performing a manual machine password change. We just retired our last 2003 DC and hope that we are not going to be surprised by this popping up again. The patch does not seem to have totally prevented it from occurring. - Anonymous
September 09, 2014
Hello,
I have the same experience, after installing the hotfix on our DC's, we still need to reboot our servers which were already affected and showing the known 'error 4' in the eventviewer... Is this the expected behavior?
I guess the hotfix is effective for future password changes of our machines?
Regards, - Anonymous
September 09, 2014
I guess I was to soon with the above post, we have other member servers who changed their computer password after applying the hotfix and are experiencing the same issue... - Anonymous
September 10, 2014
Hi guys,
I'm not able to install the fix on my W2K12 PDC and SDC because I'm getting the following error:
Impossibile installare l'aggiornamento di Windows a causa dell'errore 2149842967 "" (riga di comando: ""C:Windowssystem32wusa.exe" "C:UsersAdministrator.ABREGDesktopWindows6.1-KB2768494-x64.msu" ")
Best regards. - Anonymous
September 10, 2014
I've tried the option 3.
I've changed the GPO Policy before to install the hotfix and selected RC4_HMAC_MD5, DES_CBC_CRC, DESC_CBC_MD5 options. I'm getting the following error on PDC : "The encryption type requested is not supported by the KD error 0x80090342 " .
Regards - Anonymous
September 14, 2014
Hello All
i can just support the statement of markey165 posted on 2 Sep 2014 11:27 PM...
I guess the Hotfix should help if other Client machines are still not aware of AES right ?
btw: all my DCs in that particular Child Domain are 2012 R2 and i applied the Hotfix to all of them and still some Clients are experience the same issue as before.. just once per Client so far... think that supports the statement of others as well...
so anyone has another Idea, update on that ? i will definitely do a network trace but would like to inform that i think the hotfix alone is not solving that issue... also before 2012R2 we had not such issues at all just to let you know that i dont believe in some other issues related to KDC / Kerberos in our environment!
regards
Ueli - Anonymous
September 18, 2014
안녕하세요. 한국 마이크로소프트 플랫폼 기술지원팀 이현정 입니다. 도메인 컨틀롤러를 Windows Server 2003 에서 Windows Server 2012 R2 로 업그레이드 - Anonymous
September 19, 2014
Hi,
A month ago we removed our last W2K3 DC and since then we have had several servers with the symptoms above (Kereberos error ID 4, not able to login with domain credentials, reboot solves the problem temporarly) when they changed their computer machine password.
Two weeks ago we installed the hotfix on our W2K12R2 DC's but this didn't solve the problem since we still had servers going is this 'state' with the above symptoms. This Always happens when they change their computer machine password.
From what I have read here, my understanding was that changing the computer machine password on our systems would/should solve the issue. So as a preventive action I would change the computer machine password manually a day before the machine would do it automatically and that seemed to help.
Now I performed this action on all our servers but immediately we had some servers giong in that 'state' again...
Does this mean that the hotfix doesn't work for our problem or...
Anyone else in this situation?
Regards,
Steve. - Anonymous
September 22, 2014
I have just tried to install the fix on a 2012 R2 Standard server running in German and also got the "update is not applicable to my machine"
After I run this PowerShell command on our DomainControllers
Get-WindowsFeature domain
We see:
[X] Active Directory-Domänendienste AD-Domain-Services Installed
Any solution? - Anonymous
September 22, 2014
Back to basics. C’est la saison des projets d’upgrade des infrastructures basées sur Windows 2003. l - Anonymous
September 22, 2014
The comment has been removed - Anonymous
September 22, 2014
@para el mundo - can you share the c:windowslogscbs.log file from that machine. It is an actual German OS and not a German MUI, correct?
@markey165 - Correct, if you have no more WS 2003 servers and the passwords have been changed, you shouldn't need this fix. We are investigating all the scenarios where the hotfix didn't work when they are reported through a support case; please use support cases to report these, not blog comments (there are a lot of things to investigate in these cases; unlike para's log file above, this will not scale).
This hotfix will eventually reach all your DCs - providing you keep up with recommended patches - via:
September 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2
http://support.microsoft.com/kb/2984006 - Anonymous
September 22, 2014
Hi NedPyle
sure, here are the cbs logs
2014-09-23 09:11:51, Info CBS TI: --- Initializing Trusted Installer ---
2014-09-23 09:11:51, Info CBS TI: Last boot time: 2014-09-09 22:30:51.488
2014-09-23 09:11:51, Info CBS Starting TrustedInstaller initialization.
2014-09-23 09:11:51, Info CBS Ending TrustedInstaller initialization.
2014-09-23 09:11:51, Info CBS Starting the TrustedInstaller main loop.
2014-09-23 09:11:51, Info CBS TrustedInstaller service starts successfully.
2014-09-23 09:11:51, Info CBS No startup processing required, TrustedInstaller service was not set as autostart
2014-09-23 09:11:51, Info CBS Startup processing thread terminated normally
2014-09-23 09:11:51, Info CBS Read out cached applicability from TiLight for package: Package_for_KB2989971~31bf3856ad364e35~amd64~~6.3.1.0, ApplicableState: 80, CurrentState:0 - Anonymous
September 23, 2014
That's just the first few lines, your CBS log should be many KB in size. Can you share via a OneDrive (or similar) link? - Anonymous
September 24, 2014
The comment has been removed - Anonymous
September 24, 2014
hi guys i am facing the that when i am using ldap authentication of windows server 2008 with zyxel usg 1000 it will take more than 1 minute .how i can solve this. pls help me - Anonymous
September 24, 2014
With onedrive you can share a file with an individual via email address. My email is my name, @microsoft.com. You can also just open a support case with MS, and upload through the case management system. - Anonymous
September 25, 2014
The comment has been removed - Anonymous
September 25, 2014
@ para and any other folks having trouble installing the hotfix, where it states it is not applicable even if you have DC role installed: I took a look at the log he shared and it was that the update was already installed, probably because had already installed this roll up (which contains the fix):
September 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2
http://support.microsoft.com/kb/2984006
The hotfix message is not as clear as it could be in this scenario, but I don't have any way to make it clearer, sorry. - Anonymous
September 28, 2014
We hit two machines this morning that lost trust. New 2012 R2 DC has the Event ID 4 errors.
This is a migration but from SBS 2011 Standard with both domain and forest at 2008 R2 native.
Perhaps when SBS set itself up it was at 2003 and then we elevated both domain/forest as I recall.
For now we have updated the 2012 R2 and R&Rd the affected PCs. - Anonymous
October 04, 2014
It turns out that weird things can happen when you mix Windows Server 2003 and Windows Server 2012 R2 domain controllers - Ask the Directory Services Team - Site Home - TechNet Blogs - Anonymous
October 19, 2014
Hi, I would like to know exactly how does the hotfix resolved the issue.
For my case, after applying the patch, I enable AES 256 encryption (and disabling the rest) as part of the hardening process for the new 2012 R2 member servers in a new AD site containing only 2012 R2 DC. After this, the member servers in the new site started exhibiting the same symptoms as the one described here.
Re-enabling RC4 resolved the issue.
Does the patch force all computer account password to use RC4 encryption, so as to be compatible with W2K3 DC? - Anonymous
March 04, 2016
"Most experienced AD admins know that this can happen " - hmmmm. I know this is a mostly moot issue 18 mo. later, but, I had to chuckle a little bit, and it must have been odd for the author to write this statement.Most experienced AD admins would NEVER allow their environment to be in such a condition. A few odd 2003 "member servers" still left in the your environment? OK, i can see that..., but 2003 Domain Controllers? Really? How/Why/What? Even if "the Business" came to us and said, "we have a business case", we (any admin worth his salt) would in a nice way, say, "Sorry, bub... not a chance". I'll go create an entire new (2003) domain, if that's even possible still, and isolate that business requestor's machine/app/whatever that "Requires" not only 2003 func. level, but a 2003 domain controller?Yeah, I just don't get any corner case that would support an AD Admin working through 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012... I write those out because I want to highlight the amount of time that "most experienced" AD admins would have long before migrated to 2008, or hence 2012 let alone R2.