Windows Server 2016
A Microsoft server operating system that supports enterprise-level management updated to data storage.
2,505 questions
VM Deployment from VMM 2019 fails with SR-IOV enabled (Static IP configuration not executed)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 92%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
Ebi3
1
Reputation point
Hi everyone,
I am having trouble getting Windows Server 2016 VMs deployed from VMM 2019 if SR-IOV is enabled on the Port Profile for that VM. The hardware NIC used is a Mellanox ConnectX-4 Lx adapter which works otherwise fine for us.
I've narrowed down the issue to the [NETIOUGC.EXE] TCPIP step of the setup where things start to go wrong.
Here's the section of the setupact.log from C:\Windows\Panther\UnattendGC:
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Transition to context 2 accepted for key 'Interfaces'.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Transition to context 3 accepted for key '00-1D-D8-B7-1C-29'.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Transition to context 4 accepted for key 'Ipv4Settings'.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] Service DHCP is already running
2021-02-15 11:32:09, Error [NETIOUGC.EXE] TCPIP: Consumer 2 REJECTED the value named 'DhcpEnabled' (under 'Ipv4Settings') with status 0x2.
2021-02-15 11:32:09, Error [NETIOUGC.EXE] TCPIP: Error processing values under 'Ipv4Settings' registry key: 0x2.
2021-02-15 11:32:09, Error [NETIOUGC.EXE] TCPIP: Error while processing the the 'Ipv4Settings' registry key.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Returning from context 4 ('Ipv4Settings') to context 3 ('00-1D-D8-B7-1C-29').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Transition to context 6 accepted for key 'Routes'.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Transition to context 7 accepted for key '1'.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Consumer 6 accepted the value named 'NextHopAddress' (under '1').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Consumer 7 accepted the value named 'Metric' (under '1').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Consumer 5 accepted the value named 'Prefix' (under '1').
2021-02-15 11:32:09, Error [NETIOUGC.EXE] TCPIP: Failed to add the constructed route table entry to the stack: 0x490.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Returning from context 7 ('1') to context 6 ('Routes').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Returning from context 6 ('Routes') to context 3 ('00-1D-D8-B7-1C-29').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Transition to context 5 accepted for key 'UnicastIpAddresses'.
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Consumer 4 accepted the value named '1' (under 'UnicastIpAddresses').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Returning from context 5 ('UnicastIpAddresses') to context 3 ('00-1D-D8-B7-1C-29').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Returning from context 3 ('00-1D-D8-B7-1C-29') to context 2 ('Interfaces').
2021-02-15 11:32:09, Info [NETIOUGC.EXE] TCPIP: Returning from context 2 ('Interfaces') to context 1 ('Software\Microsoft\Windows NT\CurrentVersion\UnattendSettings\Netio').
And here's the network settings section of the unattend.xml file VMM creates:
<settings pass="specialize" wasPassProcessed="true">
<component name="Microsoft-Windows-UnattendedJoin" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Identification>
<MachineObjectOU>OU=Server,DC=domain,DC=local</MachineObjectOU>
<JoinDomain>domain.local</JoinDomain>
<Credentials>
<Domain>DOMAIN</Domain>
<Username>DomainAdmin</Username>
<Password>*SENSITIVE*DATA*DELETED*</Password>
</Credentials>
</Identification>
</component>
<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RegisteredOrganization>Organisation</RegisteredOrganization>
<ComputerName>ComputerName</ComputerName>
<ProductKey>*SENSITIVE*DATA*DELETED*</ProductKey>
</component>
<component name="Microsoft-Windows-TCPIP" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Interfaces>
<Interface wcm:action="add">
<Ipv4Settings>
<DhcpEnabled>false</DhcpEnabled>
</Ipv4Settings>
<UnicastIpAddresses>
<IpAddress wcm:action="add" wcm:keyValue="1">192.168.200.100/24</IpAddress>
</UnicastIpAddresses>
<Identifier>00-1D-D8-B7-1C-29</Identifier>
<Routes>
<Route wcm:action="add">
<Identifier>1</Identifier>
<Prefix>0.0.0.0/0</Prefix>
<NextHopAddress>192.168.200.1</NextHopAddress>
<Metric>1</Metric>
</Route>
</Routes>
</Interface>
</Interfaces>
</component>
<component name="Microsoft-Windows-DNS-Client" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Interfaces>
<Interface>
<Identifier>00-1D-D8-B7-1C-29</Identifier>
<EnableAdapterDomainNameRegistration>true</EnableAdapterDomainNameRegistration>
<DisableDynamicUpdate>false</DisableDynamicUpdate>
<DNSDomain>Domain.local</DNSDomain>
<DNSServerSearchOrder>
<IpAddress wcm:action="add" wcm:keyValue="1">192.168.200.2</IpAddress>
<IpAddress wcm:action="add" wcm:keyValue="2">192.168.200.3</IpAddress>
</DNSServerSearchOrder>
</Interface>
</Interfaces>
</component>
</settings>
Now if SR-IOV is disabled everything is working well. However since the Mellanox Drivers (some old version) are part of the default drivers of Windows Server 2016 the Virtual Function (VF) of that network adapter is already present during that setup step. As VF and vmNIC share the same MAC-Address I assume, that the unattended setup is trying to configure the VF instead of the actual vmNIC which is why this fails as interfaces are identified by their MAC-Address.
Get-NetAdapter | ft Name,InterfaceDescription,ifIndex,MacAddress
Name InterfaceDescription ifIndex MacAddress
---- -------------------- ------- ----------
Ethernet 2 Mellanox ConnectX-4 Lx Virtual Ethernet Adapter 3 00-1D-D8-B7-1C-29
Ethernet Microsoft Hyper-V Network Adapter 2 00-1D-D8-B7-1C-29
One data point indicating this might be the case is that we don't see this issue with a Broadcom Adapter that has no default driver in Windows Server 2016. Here the setup works fine and after installing the driver in the VM the SR-IOV VF works propely.
Adding an updated Mellanox driver to the VHD image doesn't work either - the issue remains. As soon as I deactivate SR-IOV in the port profile the deployment starts working. If I activate SR-IOV after the VM has been deployed the VM also works with SR-IOV enabled - so it is really is just a setup / deployment issue.
Could it be that the unattended windows setup has an issue if there are multiple NICs with the same MAC-Address, even if one of them is just a (normally invisible) VF?
Any ideas how to solve it (except switching port profiles after deployment to enable SR-IOV later)?
Thank you and best wishes,
Jens
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 30%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Crystal-MSFT 48,591 Reputation points Microsoft Vendor2021-02-16T03:30:13.063+00:00 @Ebi3 , From your description, I find the issue seems to be that VM Deployment from VMM 2019 fails with IPSettings error for Mellanox ConnectX-4 Lx adapter when SR-IOV is enabled. For Broadcom Adapter, it is working. if there's any misunderstanding, feel free to let us know.
Here, I would like your help to do a test to deploy the VM via Hyper-V using Mellanox ConnectX-4 Lx adapter with the SR-IOV enabled to see if it can success or fail.
Meanwhile, for the MAC address, in my lab, the virtual switch and the network adapter it connects to are the same.
In addition, I notice the windows server is with 2016 version, I have faced some known issue for VMM 2019 on windows server 2016. Here, I suggest to upgrade to Windows server 2019 and installed the latest updates to see if it is working.Thanks and have a nice day!
-
Ebi3 1 Reputation point
2021-02-16T12:56:51.29+00:00 I find the issue seems to be that VM Deployment from VMM 2019 fails with IPSettings error for Mellanox ConnectX-4 Lx adapter when SR-IOV is enabled. For Broadcom Adapter, it is working. if there's any misunderstanding, feel free to let us know
This is correct. As there is no Default Driver for the Broadcom VF it is not available during setup until the driver has been installed after setup. So the IP configuration through unattend.xml works.
I would like your help to do a test to deploy the VM via Hyper-V using Mellanox ConnectX-4 Lx adapter with the SR-IOV enabled to see if it can success or fail.
Could you please elaborate what exactly you mean by deploy the VM via Hyper-V? We've so far not having done any automated VM deployment outside of VMM. Or did I misunderstood you?
I suggest to upgrade to Windows server 2019
Unfortunately that's not an option currently for cost reasons.
-
Ebi3 1 Reputation point
2021-02-16T13:03:06.47+00:00 Regarding the MAC-Address - just to avoid any misunderstanding on that: After setup both Broadcom and Mellanox VFs work fine within the VM. It is just the IP configuration at unattended setup that fails for the Mellanox.
-
Crystal-MSFT 48,591 Reputation points Microsoft Vendor
2021-02-18T01:51:20.39+00:00 @Ebi3 , Thanks for your reply. After checking your information again, I find there's some misunderstanding from me before.
For the MAC address I check on my side is on Hyper-V. It seems the information on your side is on the Windows server 2016 VM. Here, I would like to confirm if MAC address on the VF of Broadcom also has the same MAC address with the "Microsoft Hyper-v Network Adapter" after installing its driver?
-
Ebi3 1 Reputation point
2021-02-18T09:06:23.527+00:00 @Crystal-MSFT , Yes, this is inside the Windows Server 2016 VM.
After installing the Broadcom VF driver inside the VM it also shares the same MAC address as the Microsoft Hyper-V Network Adapter. However before installing that driverin the VM the VF does not appear at all when running the Get-NetAdapter powershell cmdlet. It's only after installing the driver the VF shows up.
So at the time of that unattended VM setup step that fails there would be only one NetAdapter available with that MAC Address.
Thank you again and all the best,
Jens -
Crystal-MSFT 48,591 Reputation points Microsoft Vendor
2021-02-19T06:50:54.08+00:00 @Ebi3 , Thanks for the confirmation. My understanding is that when configuring the Ip address for "00-1D-D8-B7-1C-29", it is failed because both the VF adapter and Microsoft Hyper-V network adapter share the same MAC address. I think your conjecture at first is correct.
For such situation, I have two thoughts: to try
- Try to configure VM network, configure IP address pool and choose "Static IP (from a static IP pool)" in the VM template. This may skip the VF adapter.
https://video2.skills-academy.com/en-us/system-center/vmm/network-pool?view=sc-vmm-2019 - Write a script to set IP on the Hyper-V adapter via script and add it into GUIRunOnce to try if it can configure the Ip correctly:
https://www.tech-coffee.net/rename-vms-network-adapters-automatically-with-virtual-machine-manager-2016/
Note: Non-Microsoft link, just for the reference.
- Try to configure VM network, configure IP address pool and choose "Static IP (from a static IP pool)" in the VM template. This may skip the VF adapter.
-
Ebi3 1 Reputation point
2021-02-19T10:03:56.237+00:00 @Crystal-MSFT Thank you for your suggestions.
Proposal 1 is exactly what I did and what fails. VMM uses the unattend.xml file to propagate static IP addresses to the newly deployed VM. However because of the VF present with the same MAC it seems that the IP address setup step of the unattended setup fails.
Proposal 2 is kind of what I did now to solve the issue - I created a second port profile with SR-IOV disabled and use this during deployment. This way at least I keep the VMM deployment features (e.g. domain join, ip address assignment etc. which all otherwise fail since the IP config of the newly created machine is wrong).
My deployment scripts do this automatically before deployment, then deploy the VM and then change the profile of the created VM back to SR-IOV enabled. This workaround works but of course voids any GUI deployment options.
It appears this might be an oversight of the NETIOUGC.EXE process of Windows Server 2016 when identifying network adapters based on MAC?
-
Crystal-MSFT 48,591 Reputation points Microsoft Vendor
2021-02-22T02:35:56.007+00:00 @Ebi3 , I agree. It looks like the NETIOUGC.EXE process of Windows Server 2016 identify network adapters based on MAC. which cause the IP address issue. Maybe we can contact the adapter provider to see if any action can be done to avoid the conflict MAC issue or open case to see if any other action can be done to avoid the IP setting issue
https://support.microsoft.com/en-us/topic/global-customer-service-phone-numbers-c0389ade-5640-e588-8b0e-28de8afeb3f2Thanks and have a nice day!
Sign in to comment
Sign in to answer