Windows Server 2019 Hyper-V Failover Cluster .avhdx files remaining after backup

Dave Baddorf 101 Reputation points
2022-08-10T15:29:26.113+00:00

Hello!

I have a customer with a three node Windows Server 2019 Hyper-V Failover Cluster. When I backup VM's w/ Veeam, I sometimes get left-over .avhdx files that don't automatically merge. With additional backups, sometimes the VM keeps adding additional .avhdx files - other times there remains only a single .avhdx file. Hyper-V Manager doesn't show any Checkpoints. And when I try to look at the Hyper-V Settings in Failover Cluster manager, the MMC crashes with the error "MMC has detected an error in a snap-in".

I have a process to manually fix these VM's:

  • Remove VM from Cluster [this is necessary because the Failover Cluster MMC crashes and I can't edit the VM disks in Hyper-V Manager when the VM is part of a cluster]
  • Shutdown VM [this is a pain]
  • Manually merge .avhdx files in chain back to parent [Merge-VHD or AZSBTools]
  • Fix disk in Hyper-V Manager Settings back to .vhdx
  • Rejoin Cluster

The really interesting thing is that [Get-WinEvent Microsoft-Windows-Hyper-V-* | Where-Object{$_.Message -like "<VMName>"} ] shows "background disk merge has been started" and "background disk merge has been finished successfully" for the VM in question. Yet the .avhdx had been created at some point and not cleaned up automatically.

I'd love to be able to fix these VM's without having to shutdown the VM. I'd really love to be able to not have these .avhdx files remain and have the system automatically merge them.

Any suggestions? I'd certainly be glad to hear them!

Dave

Windows Server 2019
Windows Server 2019
A Microsoft server operating system that supports enterprise-level management updated to data storage.
3,563 questions
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
2,609 questions
Windows Server Clustering
Windows Server Clustering
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Clustering: The grouping of multiple servers in a way that allows them to appear to be a single unit to client computers on a network. Clustering is a means of increasing network capacity, providing live backup in case one of the servers fails, and improving data security.
971 questions

8 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Dave Baddorf 101 Reputation points
    2023-01-10T14:45:41.537+00:00

    It's been a while since I checked in and I wanted to update the status of this issue.

    While I don't have the problem resolved, it is much more manageable now. I ended up making smaller batches of VM backups in Veeam. It seems like this has helped mitigate most of the problems that I was having with leftover .avhdx disks. I still have an occasional VM that I have to take down and remove from the Cluster & Hyper-V at which point the single .avhdx merges automatically. This is much better than the several VM's per week that I was having earlier and also having multiple .avhdx files per VM disk volume which had be be manually merged.

    1 person found this answer helpful.
  2. Shaun 6 Reputation points
    2023-01-14T02:15:00.91+00:00

    Hi Dave,

    Our issue ended up being incorrect/missing permissions on the folder where the VMs were stored.

    I created a new VMs folder with correct permissions to hold VMs. I then moved the VMs to the fresh VM storage folder to regenerate permissions/ownership.

    Since completing this, I have had no issues, either with Veeam or creating checkpoints.

    Shaun

    1 person found this answer helpful.
    0 comments
  3. Shaun 6 Reputation points
    2022-08-19T00:38:16.947+00:00

    Hey Dave,

    I am also getting the same issue of extra .avhdx files with no checkpoints showing. My Hyper-V cluster is Win Server 2022. And I've just added the cluster to Veeam. My Failover Cluster manager MMC also fails when looking at VMs Settings.

    Running
    Get-VMSnapshot -ComputerName "host server" -VMName "vm name"
    doesn't show any checkpoints.
    And if I manually make a checkpoint, it doesn't show up in Hyper-V Manager.

    When you have run your fix, has the issue returned?

    Shaun

  4. Santosh Dangle 0 Reputation points
    2023-01-25T11:40:05.36+00:00

    Hello Shaun, Can you help to send the type of permissions you setup for your vm folders?

    Note: We have joined all systems to central domain server

    0 comments
  5. Shaun 6 Reputation points
    2023-01-26T22:49:22.2366667+00:00

    User's image

    Here you go Santosh.

    We have a Cluster with 3 hosts, with a cluster shared volume for VM storage. Our hosts and VMs are domain joined.

    0 comments