Hello! I have a customer with a three node Windows Server 2019 Hyper-V Failover Cluster. When I backup VM's w/ Veeam, I sometimes get left-over .avhdx files that don't automatically merge. With additional backups, sometimes the VM keeps adding additional .avhdx files - other times there remains only a single .avhdx file. Hyper-V Manager doesn't show any Checkpoints. And when I try to look at the Hyper-V Settings in Failover Cluster manager, the MMC crashes with the error "MMC has detected an error in a snap-in". I have a process to manually fix these VM's: Remove VM from Cluster [this is necessary because the Failover Cluster MMC crashes and I can't edit the VM disks in Hyper-V Manager when the VM is part of a cluster] Shutdown VM [this is a pain] Manually merge .avhdx files in chain back to parent [Merge-VHD or AZSBTools] Fix disk in Hyper-V Manager Settings back to .vhdx Rejoin Cluster The really interesting thing is that [Get-WinEvent Microsoft-Windows-Hyper-V-* | Where-Object{$_.Message -like " <VMName> "} ] shows "background disk merge has been started" and "background disk merge has been finished successfully" for the VM in question. Yet the .avhdx had been created at some point and not cleaned up automatically. I'd love to be able to fix these VM's without having to shutdown the VM. I'd really love to be able to not have these .avhdx files remain and have the system automatically merge them. Any suggestions? I'd certainly be glad to hear them! Dave

I am still periodically having this issue with the Checkpoints not merging. Today I tried to do a bit of further research and found that the vmms.exe process on the owner Hyper-V server has a .vmcx and a .vmrs file opened in the Snapshot folder of the offending VM (while the VM is shutdown). These files being opened prevent a Storage Migration of the VM to a different folder. In trying to manually move folders, etc., somehow Hyper-V lost the configuration for the VM, so I ended up doing a restore. But next time this happens, I'd like to see if these same two files (the .vmcx and the .vmrs) are open on another Hyper-V host in the cluster after a migration between hosts. I don't see any problems with permissions - it just seems like a bug in Hyper-V for the vmms.exe to not be able to automatically merge the Checkpoint as it should, but it somehow keeps these files open. My goal isn't to prevent this Checkpoint merge issue, but just to try to find a way to quickly recover from it without having to remove the VM from Failover Cluster & from Hyper-V and having to manually add it back as a "new" VM. (I could backup and restore the VM or export & import, but that would take a while).

Dear diary, I am not sure why I keep thinking that I can figure out a way to fix these checkpoints without removing the VM and re-adding it. I can't run a shared-nothing migration to another Hyper-V - the broken checkpoint won't allow the migration. I can't even run a live-migration within the cluster when the checkpoint is messed up. I can manually merge the .avhdx to the .vhdx and replace the Hard Drive in Hyper-V settings. That leaves the VM intact, but it is still broken under the hood because I can't open Settings in Failover Cluster Manger without a crash of the Cluster Manger Console. At least this doesn't work consistently. Again, I need to remove the VM and re-add. I know that no one will read this diary entry, but I guess it's helpful to get it all out. Dave

Windows Server 2019 Hyper-V Failover Cluster .avhdx files remaining after backup

Dave Baddorf 101

Hello!

I have a customer with a three node Windows Server 2019 Hyper-V Failover Cluster. When I backup VM's w/ Veeam, I sometimes get left-over .avhdx files that don't automatically merge. With additional backups, sometimes the VM keeps adding additional .avhdx files - other times there remains only a single .avhdx file. Hyper-V Manager doesn't show any Checkpoints. And when I try to look at the Hyper-V Settings in Failover Cluster manager, the MMC crashes with the error "MMC has detected an error in a snap-in".

I have a process to manually fix these VM's:

Remove VM from Cluster [this is necessary because the Failover Cluster MMC crashes and I can't edit the VM disks in Hyper-V Manager when the VM is part of a cluster]
Shutdown VM [this is a pain]
Manually merge .avhdx files in chain back to parent [Merge-VHD or AZSBTools]
Fix disk in Hyper-V Manager Settings back to .vhdx
Rejoin Cluster

The really interesting thing is that [Get-WinEvent Microsoft-Windows-Hyper-V-* | Where-Object{$_.Message -like "<VMName>"} ] shows "background disk merge has been started" and "background disk merge has been finished successfully" for the VM in question. Yet the .avhdx had been created at some point and not cleaned up automatically.

I'd love to be able to fix these VM's without having to shutdown the VM. I'd really love to be able to not have these .avhdx files remain and have the system automatically merge them.

Any suggestions? I'd certainly be glad to hear them!

Dave

Dave Baddorf 101 Reputation points

2022-08-24T19:28:24.433+00:00

Some additional information:

I did some experimenting with Veeam and even with "Guest Processing - Enable application-aware processing" disabled and with "Enable Hyper-V Guest Quiescense" disabled, Veeam still triggers the creation of a Recovery Checkpoint (apparently different than a Standard or Production Checkpoint). So it doesn't appear like I can prevent the creation of this Recovery Checkpoint .avhdx file - I need to find a way to make sure that it gets merged back in automatically.

The Get-VMSnapshot command shows this .avhdx file with a CheckpointType and SnapshotType of "Recovery" instead of "Standard".
Dave Baddorf 101 Reputation points

2022-10-04T17:16:55.783+00:00

For all of those who are interested (which may just be me):

I wrote a script to show me any .avhdx files which exist on the CSV volumes. It runs twice a day after my Veeam Backups. This way I know exactly when these .avhdx files haven't been cleaned up. For example, I had an .avhdx for VM1 show up after the 7:00PM Veeam backup on Oct 1st.

The really intriguing thing is that I can go to that Hyper-V Server's Event Log (Applications and Services Logs - Microsoft - Windows - Hyper-V-VMMS - Admin) and I see the following events:
* Event ID 19070, 7:09:30PM "VM1 Background disk merge has been started"
* Event ID 19090: 7:09:31PM "VM1 Background disk merge has been interrupted"
* Event ID 19070, 7:09:32PM "VM1 Background disk merge has been started"
* Event ID 19080, 7:09:35PM "VM1 Background disk merge has been finished successfully"

The VM's Checkpoint looks like it was properly merged. Yet the .avhdx remained.

I had another VM which left a .avhdx file after Oct 2nd's backup. It also had the disk merge started and finished without the interrupted step. You'd think that this would certainly not leave the .avhdx file behind. Yet it did.

If these VM's Checkpoints were merged successfully, why does the .avhdx files still exist and why does Hyper-V Manager Settings still show the the .avhdx drive for the VM setups?

If anyone (besides myself) has any good ideas, I am certainly willing to entertain them.

8 answers

Dave Baddorf 20 Reputation points

2024-02-05T15:49:15.7933333+00:00

I am still periodically having this issue with the Checkpoints not merging. Today I tried to do a bit of further research and found that the vmms.exe process on the owner Hyper-V server has a .vmcx and a .vmrs file opened in the Snapshot folder of the offending VM (while the VM is shutdown). These files being opened prevent a Storage Migration of the VM to a different folder. In trying to manually move folders, etc., somehow Hyper-V lost the configuration for the VM, so I ended up doing a restore. But next time this happens, I'd like to see if these same two files (the .vmcx and the .vmrs) are open on another Hyper-V host in the cluster after a migration between hosts. I don't see any problems with permissions - it just seems like a bug in Hyper-V for the vmms.exe to not be able to automatically merge the Checkpoint as it should, but it somehow keeps these files open.
My goal isn't to prevent this Checkpoint merge issue, but just to try to find a way to quickly recover from it without having to remove the VM from Failover Cluster & from Hyper-V and having to manually add it back as a "new" VM. (I could backup and restore the VM or export & import, but that would take a while).
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more
Dave Baddorf 20 Reputation points

2024-02-28T15:56:21.14+00:00

Dear diary, I am not sure why I keep thinking that I can figure out a way to fix these checkpoints without removing the VM and re-adding it. I can't run a shared-nothing migration to another Hyper-V - the broken checkpoint won't allow the migration. I can't even run a live-migration within the cluster when the checkpoint is messed up. I can manually merge the .avhdx to the .vhdx and replace the Hard Drive in Hyper-V settings. That leaves the VM intact, but it is still broken under the hood because I can't open Settings in Failover Cluster Manger without a crash of the Cluster Manger Console. At least this doesn't work consistently. Again, I need to remove the VM and re-add. I know that no one will read this diary entry, but I guess it's helpful to get it all out. Dave
Please sign in to rate this answer.
Ben Brasile 0 Reputation points

2024-06-26T20:09:02.68+00:00

Hi Dave, we've got a similar situation happening with one of the VMs in our failover cluster that's backed up with Veeam. When the AVHDX files get close to filling up the cluster storage for that VM I have to shut down the VM. It then begins auto-merging the snapshots back into the main file. The merge status is visible in the Hyper-V manager for the associated host (but no indication of merging when viewing in the Failover cluster manager). Does that not happen in your environment?

I'm still working out why the snapshots aren't being merged back in after the Veeam backups.
Sign in to comment

Share via

Windows Server 2019 Hyper-V Failover Cluster .avhdx files remaining after backup

8 answers