How do I get the VSS SQL writer to rebuild needed configuration?

Alexander Lindberg 1 Reputation point
2024-02-27T08:10:16.6066667+00:00

Hello Q & A We have a rather odd situation on our hands. A server 2016 machine was setup years ago to run our Active Directory Sync Service. Back then, the server was a regular machine without a specific role, and the ADSync Service was setup running as a local user as recommended. A few years later this machine was promoted to a DC to replace our old SBS2008-machine. In the beginning of this year we noticed that something was off, event logs showed repeated fails of the automated update process for ADSync and somewhere along trying to get it to install the sync service failed completely. After some head scratching I finally found out why (on the topic of: make something idiot proof, and the world invents a better idiot!). ADSync on a DC should not be run by a local service account, but rather a domain account. So our config had gotten illegal due to the promotion, but this did not manifest itself until a specific update requirement. I tried at first to create a new AD user for the service, and giving sysadmin role to it on the WID instance but no luck. As we have very simple sync rules, the next easiest option was to spin up a VM of an old backup of the server with a working ADSync, note down the config, and uninstall and reinstall ADSync on the current server. This broke something though, because our backups (using UrBackup) started to fail, getting event 12289 from WIDWRITER in the event logs. There is no description for this error, but it includes this:

RegOpenKeyExW(FFFFFFFF80000003,[SID goes here]\Software\Microsoft\Microsoft SQL Server\UserInstances,...)
0x80070005

This is under HKEY_USERS, the SID belongs to the domain administrator, the UserInstances key exists but it is empty. The same key exists under the new domain user created to run ADSync. It contains two subkeys, one for the ADSync 2019 instance (LOCALDB#SHF48841) and one for the MSSQLLocalDB (LOCALDB#84A994F6). In SQLWriterLogger.txt I can find this block after a failed backup:

[02/23/2024 00:04:28, TID 105c] ****************************************************************
[02/23/2024 00:04:28, TID 105c] **  VSS SQL BACKUP BEGIN - ID: 105c
[02/23/2024 00:04:28, TID 105c] ****************************************************************
[02/23/2024 00:04:28, TID 105c] Component based backup selected.
[02/23/2024 00:04:28, TID 105c] Database count from metadata is 4
[02/23/2024 00:04:28, TID 105c] Database master on instance RADIO1\LOCALDB#SHF48841 found in metadata
[02/23/2024 00:04:28, TID 105c] Database model on instance RADIO1\LOCALDB#SHF48841 found in metadata
[02/23/2024 00:04:28, TID 105c] Database msdb on instance RADIO1\LOCALDB#SHF48841 found in metadata
[02/23/2024 00:04:28, TID 105c] Database ADSync on instance RADIO1\LOCALDB#SHF48841 found in metadata
[02/23/2024 00:04:28, TID 105c] Backup type is VSS_BT_FULL
[02/23/2024 00:04:40, TID 105c] Entering SQL Writer OnPrepareSnapshot.
[02/23/2024 00:04:40, TID 105c] SQLInstanceNames Registry was not found
[02/23/2024 00:04:40, TID 105c] Enumerating Local DB User Instances
[02/23/2024 00:04:40, TID 3814] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 4108] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 4108] SQLSTATE:	42000	Native Error:	3013
[02/23/2024 00:04:40, TID 3814] SQLSTATE:	42000	Native Error:	3013
[02/23/2024 00:04:40, TID 3814] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 4108] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 3814] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 3814] Error message:	BACKUP DATABASE is terminating abnormally.
[02/23/2024 00:04:40, TID 3814] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 3814] SQLSTATE:	42000	Native Error:	18210
[02/23/2024 00:04:40, TID 3814] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 4108] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 4108] Error message:	BACKUP DATABASE is terminating abnormally.
[02/23/2024 00:04:40, TID 4108] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 1fe4] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 1fe4] SQLSTATE:	42000	Native Error:	3013
[02/23/2024 00:04:40, TID 3814] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 3814] Error message:	BackupVirtualDeviceSet::Initialize: CoCreateInstance failure on backup device '{638784E3-54FF-4525-8F12-702CFFB0594E}2'. Operating system error 0x800703fa(Illegal operation attempted on a registry key that has been marked for deletion.).
[02/23/2024 00:04:40, TID 1fe4] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 4108] SQLSTATE:	42000	Native Error:	18210
[02/23/2024 00:04:40, TID 3814] SqlConnection::ExecCommand: Error calling ICommandText::Execute.  hr = 0x80040e14.	SQLSTATE: 42000, Native Error: 3013	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BACKUP DATABASE is terminating abnormally.	SQLSTATE: 42000, Native Error: 18210	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BackupVirtualDeviceSet::Initialize: CoCreate
[02/23/2024 00:04:40, TID 3814] SqlConnection::ExecCommand: Throwing HRESULT code 0x8000ffff. Previous HRESULT code = 0x80040e14
[02/23/2024 00:04:40, TID 4108] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 1fe4] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 1fe4] Error message:	BACKUP DATABASE is terminating abnormally.
[02/23/2024 00:04:40, TID 1fe4] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 1fe4] SQLSTATE:	42000	Native Error:	18210
[02/23/2024 00:04:40, TID 4108] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 4108] Error message:	BackupVirtualDeviceSet::Initialize: CoCreateInstance failure on backup device '{638784E3-54FF-4525-8F12-702CFFB0594E}1'. Operating system error 0x800703fa(Illegal operation attempted on a registry key that has been marked for deletion.).
[02/23/2024 00:04:40, TID 1fe4] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 1fe4] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 1fe4] Error message:	BackupVirtualDeviceSet::Initialize: CoCreateInstance failure on backup device '{638784E3-54FF-4525-8F12-702CFFB0594E}3'. Operating system error 0x800703fa(Illegal operation attempted on a registry key that has been marked for deletion.).
[02/23/2024 00:04:40, TID 4108] SqlConnection::ExecCommand: Error calling ICommandText::Execute.  hr = 0x80040e14.	SQLSTATE: 42000, Native Error: 3013	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BACKUP DATABASE is terminating abnormally.	SQLSTATE: 42000, Native Error: 18210	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BackupVirtualDeviceSet::Initialize: CoCreate
[02/23/2024 00:04:40, TID 4108] SqlConnection::ExecCommand: Throwing HRESULT code 0x8000ffff. Previous HRESULT code = 0x80040e14
[02/23/2024 00:04:40, TID 1fe4] SqlConnection::ExecCommand: Error calling ICommandText::Execute.  hr = 0x80040e14.	SQLSTATE: 42000, Native Error: 3013	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BACKUP DATABASE is terminating abnormally.	SQLSTATE: 42000, Native Error: 18210	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BackupVirtualDeviceSet::Initialize: CoCreate
[02/23/2024 00:04:40, TID 1fe4] SqlConnection::ExecCommand: Throwing HRESULT code 0x8000ffff. Previous HRESULT code = 0x80040e14
[02/23/2024 00:04:40, TID 4108] HRESULT EXCEPTION CAUGHT: hr: 0x8000ffff
[02/23/2024 00:04:40, TID 1fe4] HRESULT EXCEPTION CAUGHT: hr: 0x8000ffff
[02/23/2024 00:04:40, TID 3814] HRESULT EXCEPTION CAUGHT: hr: 0x8000ffff
[02/23/2024 00:04:40, TID 2e4c] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 2e4c] SQLSTATE:	42000	Native Error:	3013
[02/23/2024 00:04:40, TID 2e4c] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 2e4c] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 2e4c] Error message:	BACKUP DATABASE is terminating abnormally.
[02/23/2024 00:04:40, TID 2e4c] HRESULT:	0X80040E14
[02/23/2024 00:04:40, TID 2e4c] SQLSTATE:	42000	Native Error:	18210
[02/23/2024 00:04:40, TID 2e4c] Error state:	1	Severity:	16
[02/23/2024 00:04:40, TID 2e4c] Source:		Microsoft SQL Server Native Client 11.0
[02/23/2024 00:04:40, TID 2e4c] Error message:	BackupVirtualDeviceSet::Initialize: CoCreateInstance failure on backup device '{638784E3-54FF-4525-8F12-702CFFB0594E}4'. Operating system error 0x800703fa(Illegal operation attempted on a registry key that has been marked for deletion.).
[02/23/2024 00:04:40, TID 2e4c] SqlConnection::ExecCommand: Error calling ICommandText::Execute.  hr = 0x80040e14.	SQLSTATE: 42000, Native Error: 3013	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BACKUP DATABASE is terminating abnormally.	SQLSTATE: 42000, Native Error: 18210	Error state: 1, Severity: 16	Source: Microsoft SQL Server Native Client 11.0	Error message: BackupVirtualDeviceSet::Initialize: CoCreate
[02/23/2024 00:04:40, TID 2e4c] SqlConnection::ExecCommand: Throwing HRESULT code 0x8000ffff. Previous HRESULT code = 0x80040e14
[02/23/2024 00:04:40, TID 2e4c] HRESULT EXCEPTION CAUGHT: hr: 0x8000ffff
[02/23/2024 00:04:41, TID 105c] HRESULT EXCEPTION CAUGHT: hr: 0x8000ffff
[02/23/2024 00:04:41, TID 105c] Freeze2000::WaitForPrepare: rethrowing
[02/23/2024 00:04:41, TID 105c] Freeze2000::WaitForPrepare: Throwing HRESULT code 0x8000ffff. Previous HRESULT code = 0x8000ffff
[02/23/2024 00:04:41, TID 105c] HRESULT EXCEPTION CAUGHT: hr: 0x8000ffff
[02/23/2024 00:04:41, TID 105c] Entering SQL Writer OnAbort.
[02/23/2024 00:04:50, TID 105c] Entering SQL Writer OnIdentify.
[02/23/2024 00:04:50, TID 105c] SQLInstanceNames Registry was not found
[02/23/2024 00:04:50, TID 105c] Enumerating Local DB User Instances
[02/23/2024 00:04:52, TID 105c] Entering SQL Writer OnPrepareBackup.

This corresponds well with what I see in the event log, but the questions are then:

  • Why/how could this have happened, why wouldn't the WID writer pick up on the new configuration?
  • Can I get the SQL Writer to rebuild the information about the WID to remediate this error, or am I looking at yet an attempt to uninstall and reinstall the ADSync service?

Also, I don't see anthything about WSUS (which is also using the WID) anywhere, which is running on this server as well, and seems to be running fine.

Any information or suggestions are greatly appreciated Best regards Alexander

SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
13,200 questions
Windows Server Backup
Windows Server Backup
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Backup: A duplicate copy of a program, a disk, or data, made either for archiving purposes or for safeguarding valuable files from loss should the active copy be damaged or destroyed.
464 questions
0 comments No comments
{count} votes