Windows Server 2016 / Hyper-V: SR-IOV Guest Teaming and Switch Embedded Teaming (SET) on Host - Best Practise

Ebi3 1 Reputation point
2021-02-17T10:14:55.36+00:00

Hi everyone,

I came accross a question that I couldn't find any updated advice on so I'd love to read your thoughts on this:

Setup is a Hyper-V-Host-Cluster with 2x 10GB-RDMA/SR-IOV-capable NICs for each host that are teamed using Hyper-V Switch Embedded Teaming. The parent partition is using two vNICs for Live Migration via RDMA which works beautifully. Managment and cluster traffic are separate Hardware NICs.

Now deploying High-Availability VMs also works nicely but as SR-IOV is used only one of the NICs is in use within any given VM.

The idea came up to use two SR-IOV vmNICs per VM and team them within the VM using LBFO (Switch Independent/Address Hashes). Both vmNICs would be connected however to the same SET-Switch on the host.
Testing this setup in the lab worked nicely except that WinRM connectivity was lost (we regained that after enabling the WMI DCOM-In Firewall Rule in the VM - any thoughts on why this was now necessary?).
However whatever advice I can find online shows VM gueest teaming only for two separate virtual switches on the host, never the same. However those advisories also are usually from a couple of years back and don't seem to feature SET on the host but rather regular Virtual Switches.

So is this a supported configuration using VM SR-IOV Guest Teaming on the same SET-Switch on the host? Is this best practice when using SR-IOV and SET simultanously? Is there any trouble on the horizon doing a setup like that? Or is it useless effort as SET will handle everything even with just one vmNIC present in the VMs?

Thank you for your thoughts and best wishes,
Jens

Windows Server 2016
Windows Server 2016
A Microsoft server operating system that supports enterprise-level management updated to data storage.
2,505 questions
System Center Virtual Machine Manager
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
2,706 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Mico Mi 1,926 Reputation points
    2021-02-18T05:45:24.113+00:00

    Hi,
    If you tested this setup nicely, it may be a supported configuration.
    Since I have not done such test yet, I hope someone with similar issues could provide some advice and ideas here.
    Maybe you can check the following docs (also from a couple of years back) and I hope they could help at your side:
    Hyper-V Host Teaming vs Guest SR-IOV Teaming
    Windows Server 2016 NIC and Switch Embedded Teaming User Guide

    Thanks for your time!
    Best Regards,
    Mico Mi

    -----------------------------

    If the Answer is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    0 comments
  2. Ebi3 1 Reputation point
    2021-02-18T09:38:19.957+00:00

    Hi,
    Thank you for your suggestions and the literature links.

    The first one unfortunately doesn't cover Switch Embedded Teaming (SET).

    The second one indicates the following hint:

    Only teams where each of the team members is connected to a different external Hyper-V switch are supported.

    However the whole section seems to be written für Windows Server 2012 R2 and doesn't mention SET on the host explicitly. So it seems it doesn't cover the scenario above. Of course I could be reading the document wrong.

    I did some performance testing with ctsTraffic.exe comparing the teamed setup with the setup that only had one vmNIC (and therefore only one VF) available and got some interesting results.

    I used the teamed/unteamed VM which I wanted to compare as ctsTraffic server (running two instances of the program at different ports with a push/pull pattern) and two different VMs (on different Hosts) as clients. Hardware and host setup was identical on all hosts. When running this against the teamed VM I got slower results both for traffic from a single VM (somewhat expected due to the teaming overhead) but also much slower when running this from two VMs which is where the teamed solution was supposed to shine. The single NIC VM on the other hand gave great results for single traffic (if my math is correctly it squeezed 40 GB of data (20GB read, 20GB write) through that line in just over 18s. It took over 26s for the teamed VM. And from two VMs the time pretty much doubled to 40s for the non-teamed VM while it spiked to 75s for the teamed VM. I repeated the tests a couple of times but the results remained similar.

    I also did some testing for SMB performance using diskspd.exe which gave no performance advantage for the teamed solution. Thanks to SMB Multichannel that's not that surprising but after removing one of the vmNICs (leaving only one vmNIC on the VM the results remained identical). That made me wonder so I've checked CPU utilization and this spiked to 100% if having two transfers in parallel (about 80% for system, 15-20% for interrupt processing). I thought this would be different when using SR-IOV but it's not - so either something is wrong or I misunderstood the concept (most likely). So CPU utilization is the limiting factor here (as widely published) making this approach not feasible to increase throughput beyond around 7.5 Gbps. I guess Guest RDMA inside the VM would be the next logical step but unfortuantely I haven't gotten this to work in Windows Server 2016 (it always results in Bluescreens on the Host).

    I've also just checked resiliency by deactivating the host NICs in the middle of a transfer. Here also the solutions with only 1 vmNIC performed better. There was a short drop in transmission speed when deactivating the primary NIC but then it went back to the 7.5 Gbps in an instant. When testing the same on the VM with the Team the transfer speed dropped to around 4 Gbps until the transfer was fully done.

    My takeaway from this is that I'll stick with a single SR-IOV-vmNIC per VM when using Switch Embedded Teaming on the host. At least until I get Guest RDMA to work...

    Any ideas on the above? Any wrong assumptions or stupid ideas in my tests above?

    Best wishes,
    Jens

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.