Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Physical)
This post is a part of the nine-part “ What’s New in Windows Server & System Center 2012 R2 ” series that is featured on Brad Anderson’s In the Cloud blog. Today’s blog post covers Windows Server 2012 R2 Storage and how it applies to the larger topic of “Transform the Datacenter.” To read that post and see the other technologies discussed, read today’s post: “What’s New in 2012 R2: IaaS Innovations.”
1) Overview
In this document, I am sharing all the steps I used to create a Windows Server 2012 R2 File Server environment, so you can try the new features like SMB Scale-Out Rebalancing and Shared VHDX for Guest Clustering. This configuration uses seven physical computers and a JBOD for shared storage. If you're not familiar with these technologies, I would strong encourage you to review the TechEd 2013 talks on Hyper-V over SMB (Understanding the Hyper-V over SMB Scenario, Configurations, and End-to-End Performance) and Shared VHDX (second half of Application Availability Strategies for the Private Cloud).
The demo setup includes the following:
- 1 DNS / Domain Controller
- 1 JBOD with SAS disks (suggested at least 4 HDDs with 250GB each)
- 3 Scale-Out File Server Cluster nodes
- 2 Hyper-V cluster nodes
- 1 VMM Server
Here’s a diagram of the setup:
Here are the details about the names, roles and IP addresses for each of the computers involved:
| Computer | Role | Net1 (Corp) | Net2 (DC/DNS) | NetR1 (RDMA) |
| JOSE-D | DNS, DC | DHCP | 192.168.100.1/24 | 192.168.101.1/24 |
| JOSE-A1 | Cluster A Node 1 | DHCP | 192.168.100.11/24 | 192.168.101.11/24 |
| JOSE-A2 | Cluster A Node 2 | DHCP | 192.168.100.12/24 | 192.168.101.12/24 |
| JOSE-A3 | Cluster A Node 3 | DHCP | 192.168.100.13/24 | 192.168.101.13/24 |
| JOSE-A | Cluster A CNO | N/A | 192.168.100.19/24 | 192.168.101.19/24 |
| JOSE-F | Scale-Out File Server DNN | N/A | N/A | N/A |
| JOSE-B1 | Cluster B Node 1 | DHCP | 192.168.100.21/24 | 192.168.101.21/24 |
| JOSE-B2 | Cluster B Node 2 | DHCP | 192.168.100.22/24 | 192.168.101.22/24 |
| JOSE-B | Cluster B CNO | N/A | 192.168.100.29/24 | 192.168.101.29/24 |
| JOSE-V | VMM Server | DHCP | 192.168.100.31/24 | 192.168.101.31/24 |
| JOSE-X1 | Cluster X, VM Node 1 | DHCP | 192.168.100.41/24 | |
| JOSE-X2 | Cluster X, VM Node 2 | DHCP | 192.168.100.42/24 | |
| JOSE-X | Guest Cluster X CNO | DHCP | 192.168.100.49/24 | |
| JOSE-XS | SQL Server, Cluster X | DHCP | 192.168.100.48/24 |
Following these steps will probably require a few hours of work end-to-end, but it is a great way to experiment with a large set of Microsoft technologies in or related to Windows Server 2012 R2, including:
- Hyper-V
- Networking
- Domain Name Services (DNS)
- Active Directory Domain Services (AD-DS)
- Shared VHDX
- Storage Spaces
- Failover Clustering
- File Servers
- PowerShell
- SQL Server
- Virtual Machine Manager
2) Required Hardware and Software, plus Disclaimers
You will need the following hardware to perform the steps described here:
- Seven computers capable of running the Windows Server 2012 R2 Preview (three of them capable of running Hyper-V).
- Virtualization settings enabled in the BIOS for the three computers that will be running Hyper-V. For details, check this blog post: https://blogs.technet.com/b/iftekhar/archive/2010/08/09/enable-hardware-settings-in-bios-to-run-hyper-v.aspx and https://technet.microsoft.com/en-us/library/cc731898.aspx
- For this setup, each computer should have at least 4GB of RAM (8GB of RAM is recommended, especially for the computer running SQL and VMM).
- An 8GB USB stick, if you’re installing Windows Server from USB and copying the downloaded software around.
You will need the following software to perform the steps described here:
- Windows Server 2012 R2 Preview from https://technet.microsoft.com/en-US/evalcenter/dn205286.aspx
- System Center VMM 2012 Preview from https://technet.microsoft.com/en-US/evalcenter/dn205295
- SQL Server 2012 SP1 Evaluation from https://www.microsoft.com/sqlserver/en/us/get-sql-server/try-it.aspx
- SQLIO, as described at https://blogs.technet.com/b/JOSE/archive/2013/03/25/sqlio-powershell-and-storage-performance-measuring-iops-throughput-and-latency-for-both-local-disks-and-smb-file-shares.aspx
Notes and disclaimers:
- A certain familiarity with Windows administration and configuration is assumed. If you're new to Windows, this document is not for you. Sorry...
- If you are asked a question or required to perform an action that you do not see described in these steps, go with the default option.
- There are usually several ways to perform a specific configuration or administration task. What I describe here is one of those many ways. It's not necessarily the best way, just the one I personally like best at the moment.
- For the most part, I use PowerShell to configure the systems. You can also use a graphical interface instead, but I did not describe those steps here.
- The JBOD used must be certified for use with Storage Spaces. For a list of certified JBODs, check the Windows Server Catalog at https://www.windowsservercatalog.com/results.aspx?bCatID=1573&cpID=0&avc=10&OR=1
- The Storage Spaces configuration shown here uses only HDDs. You could enhance it to use both SSDs and HDDs, in order to leverage new Storage Spaces features like Tiering or Write-back Cache.
- The configuration uses three file servers to demonstrate the new SMB Scale-Out Balancing capabilities when adding a node to a two-node file server cluster. It uses only two Hyper-V nodes because that’s the minimum required for a fault-tolerant Shared VHDX deployment. Typical setups will use a two-node file server cluster and a larger number of Hyper-V cluster nodes.
3) Summarized instructions for experts
If you are already familiar with Failover Clustering, Scale-Out File Servers and Hyper-V, here’s a short introduction to what it takes to configure a Shared VHDX environment with a Scale-Out File Server. A detailed step-by-step is available in the following sections of this document.
- Configure a three-node Scale-Out File Server Cluster using Windows Server 2012 R2 and Clustered Storage Spaces with Shared SAS disks.
- Configure a Hyper-V Failover cluster using Windows Server 2012 R2 with SMB shared storage as you normally would.
- Create an OS VHD or VHDX file as you would regularly, on an SMB file share. Create your VMs as you would regularly would, using the OS VHD or VHDX file. Both Generation 1 and Generation 2 VMs are supported.
- Create your VHDX data files to be shared as fixed-size or dynamically expanding, on a Clustered Shared Volume. Old VHD files or differencing disks are not supported with Shared VHDX.
- Add the shared VHDX data files to multiple VMs, using the Add-VMHardDiskDrive and the “-ShareVirtualDisk” option. If using the GUI, check the box for “Share virtual disk” when adding the VHDX data files to the VM.
- Inside the VM, install Windows Server 2012 R2 or Windows Server 2012. Make sure to install the Hyper-V integration components.
- Validate and configure the cluster inside the VMs as you normally would. The guest cluster can use classic cluster disks or Cluster Shared Volumes. Any cluster role should work as it would in a physical machine with shared SAS storage.
The rest of this document contains detailed step-by-step instructions for each of the items outlined above.
4) Configure JOSE-D (DNS, DC)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-D -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature AD-Domain-Services, RSAT-ADDS, RSAT-ADDS-Tools
# Rename DHCP network adapter to Net1, private to Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.1 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Create AD forest, reboots at the end
Install-ADDSForest -CreateDNSDelegation:$false -DatabasePath "C:\Windows\NTDS" -DomainMode "Win2012" -DomainName "JOSE.TEST" -DomainNetBIOSName "JOSE" -ForestMode "Win2012" -InstallDNS:$true -LogPath "C:\Windows\NTDS" -SYSVOLPath "C:\Windows\SYSVOL"
5) Configure JOSE-A1 (File Server Cluster A, Node 1)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A1 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.11 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Configure Storage Spaces. Assumes JBOD with 4 x 250GB HDDs = 1TB total raw. 2 mirror columns. 80GB spaces.
# Adjust the size of the volumes and the number of mirror columns based on the actual number of HDDs you have.
$s = Get-StorageSubSystem -FriendlyName *Spaces*
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $s.FriendlyName -PhysicalDisks (Get-PhysicalDisk -CanPool $true)
Set-ResiliencySetting -Name Mirror -NumberofColumnsDefault 2 -StoragePool (Get-StoragePool -FriendlyName Pool1)
New-VirtualDisk -FriendlyName Space1 -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 1GB
2..7 | % { New-VirtualDisk -FriendlyName Space$_ -StoragePoolFriendlyName Pool1 -ResiliencySettingName Mirror –Size 80GB }
1..7 | % {
$Letter ="PQRSTUV”[($_-1)]
$Number = (Get-VirtualDisk -FriendlyName Space$_ | Get-Disk).Number
Set-Disk -Number $Number -IsReadOnly 0
Set-Disk -Number $Number -IsOffline 0
Initialize-Disk -Number $Number -PartitionStyle MBR
New-Partition -DiskNumber $Number -DriveLetter $Letter -UseMaximumSize
Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart
6) Configure JOSE-A2 (File Server Cluster A, Node 2)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A2 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.12 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart
7) Configure JOSE-A3 (File Server Cluster A, Node 3)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-A3 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.13 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart
8) Configure File Server Cluster JOSE-A
# Validate cluster
Test-Cluster -Node JOSE-A1, JOSE-A2, JOSE-A3
# Create cluster
New-Cluster –Name JOSE-A -Node JOSE-A1, JOSE-A2, JOSE-A3
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "NetR1"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
(Get-ClusterNetwork NetR1).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1
# Remove default DHCP-based IP address for JOSE-A and add 2 IP addresses on 100/101 networks
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force
Add-ClusterResource -Name "Cluster IP Address 100" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 100" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.19"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 100"
Add-ClusterResource -Name "Cluster IP Address 101" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 101" | Set-ClusterParameter -Multiple @{ “Network”=”NetR1”; "Address"="192.168.101.19"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 101"
Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency "[Cluster IP Address 100] OR [Cluster IP Address 101] "
Start-ClusterResource "Cluster Name"
# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"
# Add remaining disks to Cluster Shared Volumes
Get-ClusterResource | ? OwnerGroup -eq "Available Storage" | Add-ClusterSharedVolume
# Create Scale-Out File Server
Add-ClusterScaleOutFileServerRole -Name JOSE-F
9) Configure JOSE-B1 (Hyper-V Host Cluster B, Node 1)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-B1 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart
# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
New-VMSwitch -NetAdapterName Net1 -Name Net1
Rename-NetAdapter -Name "vEthernet (Net1)" -NewName VNet1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
New-VMSwitch -NetAdapterName Net2 -Name Net2
Rename-NetAdapter -Name "vEthernet (Net2)" -NewName VNet2
Set-NetIPInterface -InterfaceAlias VNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias VNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias VNet2 -IPAddress 192.168.100.21 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias VNet2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.21 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart
10) Configure JOSE-B2 (Hyper-V Host Cluster B, Node 2)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-B2 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
Install-WindowsFeature Hyper-V, Hyper-V-PowerShell, Hyper-V-Tools -Restart
# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
New-VMSwitch -NetAdapterName Net1 -Name Net1
Rename-NetAdapter -Name "vEthernet (Net1)" -NewName VNet1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
New-VMSwitch -NetAdapterName Net2 -Name Net2
Rename-NetAdapter -Name "vEthernet (Net2)" -NewName VNet2
Set-NetIPInterface -InterfaceAlias VNet2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias VNet2 -Confirm:$false
New-NetIPAddress -InterfaceAlias VNet2 -IPAddress 192.168.100.22 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias VNet2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.22 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) -Restart
11) Configure Hyper-V Host Cluster JOSE-B
# Validate cluster
Test-Cluster -Node JOSE-B1, JOSE-B2
# Create cluster
New-Cluster –Name JOSE-B -Node JOSE-B1, JOSE-B2
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.101.*" }).Name = "NetR1"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
(Get-ClusterNetwork NetR1).Role = 3
(Get-Cluster).UseClientAccessNetworksForSharedVolumes=1
# Remove default DHCP-based IP address for JOSE-A and add 2 IP addresses on 100/101 networks
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force
Add-ClusterResource -Name "Cluster IP Address 100" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 100" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.29"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 100"
Add-ClusterResource -Name "Cluster IP Address 101" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address 101" | Set-ClusterParameter -Multiple @{ “Network”=”NetR1”; "Address"="192.168.101.29"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address 101"
Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency "[Cluster IP Address 100] OR [Cluster IP Address 101] "
Start-ClusterResource "Cluster Name"
# Create Share for VMs (run from JOSE-A1)
1..6 | % {
MD C:\ClusterStorage\Volume$_\Share
New-SmbShare -Name SHARE$_ -Path C:\ClusterStorage\Volume$_\Share -FullAccess JOSE.Test\Administrator, JOSE.Test\JOSE-B1$, JOSE.Test\JOSE-B2$, JOSE.Test\JOSE-B$
Set-SmbPathAcl -ShareName SHARE$_
}
# Create Share for File Share Witness (run from JOSE-A1)
MD C:\ClusterStorage\Volume6\Witness
New-SmbShare -Name Witness -Path C:\ClusterStorage\Volume6\Witness -FullAccess JOSE.Test\Administrator, JOSE.Test\JOSE-B$
Set-SmbPathAcl -ShareName Witness
# Configure JOSE-B Cluster with a File Share Witness (run from JOSE-B1)
Set-ClusterQuorum -NodeAndFileShareMajority \\JOSE-F.JOSE.TEST\Witness
12) Configure VMs on the Hyper-V Host Cluster JOSE-B
# Create VHD files for VMs (run on JOSE-B1)
New-VHD -Path \\JOSE-F.JOSE.TEST\Share1\VM1OS.VHDX -Fixed -SizeBytes 40GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share2\VM2OS.VHDX -Fixed -SizeBytes 40GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX -Fixed -SizeBytes 1GB
New-VHD -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX -Fixed -SizeBytes 10GB
# Create VM1 (run on JOSE-B1)
New-VM -Path \\JOSE-F.JOSE.TEST\Share1 -Name VM1 -VHDPath \\JOSE-F.JOSE.TEST\Share1\VM1OS.VHDX -SwitchName Net1 -Memory 2GB
Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Add-VMNetworkAdapter -VMName VM1 -SwitchName Net2
Add-VMHardDiskDrive -VMName VM1 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX -ShareVirtualDisk
Add-VMHardDiskDrive -VMName VM1 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX -ShareVirtualDisk
Set-VMDvdDrive -VMName VM1 -Path D:\WindowsServer2012.ISO
Start-VM VM1
Add-VMToCluster VM1
# Create VM2 (run on JOSE-B2)
New-VM -Path \\JOSE-F.JOSE.TEST\Share2 -Name VM2 -VHDPath \\JOSE-F.JOSE.TEST\Share2\VM2OS.VHDX -SwitchName Net1 -Memory 2GB
Get-VMProcessor * | Set-VMProcessor -CompatibilityForMigrationEnabled 1
Add-VMNetworkAdapter -VMName VM2 -SwitchName Net2
Add-VMHardDiskDrive -VMName VM2 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Witness.VHDX -ShareVirtualDisk
Add-VMHardDiskDrive -VMName VM2 -Path \\JOSE-F.JOSE.TEST\Share3\VM12Data.VHDX -ShareVirtualDisk
Set-VMDvdDrive -VMName VM2 -Path D:\WindowsServer2012.ISO
Start-VM VM2
Add-VMToCluster VM2
13) Configure JOSE-X1 (SQL Server Guest Cluster X, Node 1)
# Preparation steps: Install WS2012R2, rename computer, install Hyper-V IC
Rename-Computer -NewName JOSE-X1 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
# Rename the two virtual ports as Net1 and Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
Get-NetAdapter Ethernet* | Rename-NetAdapter -NewName Net2
# Configure Net2 with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.41 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Configure 2 shared disks
1..2 | % {
$Letter ="JK"[($_-1)]
Set-Disk -Number $_ -IsReadOnly 0
Set-Disk -Number $_ -IsOffline 0
Initialize-Disk -Number $_ -PartitionStyle MBR
New-Partition -DiskNumber $_ -DriveLetter $Letter -UseMaximumSize
Initialize-Volume -DriveLetter $Letter -FileSystem NTFS -Confirm:$false
}
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart
14) Configure JOSE-X2 (SQL Server Guest Cluster X, Node 2)
# Preparation steps: Install WS2012R2, rename computer, install Hyper-V IC
Rename-Computer -NewName JOSE-X2 -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
# Rename the two virtual ports as Net1 and Net2
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
Get-NetAdapter Ethernet* | Rename-NetAdapter -NewName Net2
# Configure Net2 with a static IP address for DNS / DC
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.42 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Shared Disks already configured on the first node
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart
15) Configure SQL Server Guest Cluster JOSE-X
# Validate cluster
Test-Cluster -Node JOSE-X1, JOSE-X2
# Create cluster
New-Cluster –Name JOSE-X -Node JOSE-X1, JOSE-X2
# Rename and configure networks
(Get-ClusterNetwork | ? {$_.Address -notlike "192.*" }).Name = "Net1"
(Get-ClusterNetwork | ? {$_.Address -like "192.168.100.*" }).Name = "Net2"
(Get-ClusterNetwork Net1).Role = 1
(Get-ClusterNetwork Net2).Role = 3
# Remove default DHCP-based IP address for JOSE-V and add IP address on 100 network
Stop-ClusterResource "Cluster Name"
Get-ClusterResource | ? { (($_.ResourceType -like "*Address*") -and ($_.OwnerGroup -eq "Cluster Group")) } | Remove-ClusterResource –Force
Add-ClusterResource -Name "Cluster IP Address" -Group "Cluster Group" -ResourceType "IP Address"
Get-ClusterResource –Name "Cluster IP Address" | Set-ClusterParameter -Multiple @{ “Network”=”Net2”; "Address"="192.168.100.49"; ”SubnetMask”=”255.255.255.0”; "EnableDhcp"=0 }
Get-ClusterResource “Cluster Name” | Add-ClusterResourceDependency –Resource "Cluster IP Address"
Start-ClusterResource "Cluster Name"
# Rename Witness Disk
$w = Get-ClusterResource | ? { $_.OwnerGroup -eq "Cluster Group" -and $_.ResourceType -eq "Physical Disk"}
$w.Name = "WitnessDisk"
# Install SQL Server on nodes X1 and X2
# Use IP address 192.168.100.48 for the SQL Server group
16) Configure JOSE-V (VMM Server)
# Preparation steps: Install WS2012R2, rename computer
Rename-Computer -NewName JOSE-V -Restart
# Install required roles and features, restarts at the end
Install-WindowsFeature File-Services, FS-FileServer, Failover-Clustering
Install-WindowsFeature RSAT-Clustering -IncludeAllSubFeature
# Rename DHCP network adapter to Net1
Get-NetIPAddress -PrefixOrigin DHCP | Get-NetAdapter | Rename-NetAdapter -NewName Net1
# Configure Net2 with a static IP address for DNS / DC
Get-NetAdapter Eth* | ? Status -eq Up | ? InterfaceDescription -notmatch "Mellanox*" | Rename-NetAdapter -NewName Net2
Set-NetIPInterface -InterfaceAlias Net2 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias Net2 -Confirm:$false
New-NetIPAddress -InterfaceAlias Net2 -IPAddress 192.168.100.31 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias Net2 -ServerAddresses 192.168.100.1
# Configure NetR1 with a static IP address for the RDMA network
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -eq Up | Rename-NetAdapter -NewName NetR1
Set-NetIPInterface -InterfaceAlias NetR1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias NetR1 -Confirm:$false
New-NetIPAddress -InterfaceAlias NetR1 -IPAddress 192.168.101.31 -PrefixLength 24
Set-DnsClientServerAddress -InterfaceAlias NetR1 -ServerAddresses 192.168.100.1
# Disable all disconnected adapters
Get-NetAdapter -InterfaceDescription "*IPoIB*" | ? Status -ne Up | Rename-NetAdapter -NewName NetRX
Get-NetAdapter | ? Status -ne Up | Disable-NetAdapter -Confirm:$false
# Join domain, restart the machine
Add-Computer -DomainName JOSE.TEST -Credential (Get-Credential) –Restart
# Install .NET Framework
Install-WindowsFeature NET-Framework-Core
# Install Windows Server 2012 R2 ADK
D:\adk\adksetup.exe
# Select only the “Deployment Tools” and “Windows PE” options
# Install SQL Server 2012 SP1
# New SQL Server standalone installation, Feature Selection: Database Engine Services
# Install VMM 2012 R2
# Features selected to be added: VMM management server, VMM console
# Database: VirtualManagerDB database will be created on JOSE-V
# Service Account: Local System account
17) Running the Scale-Out demo (from JOSE-V)
# Prepare - Map the SMB shares
1..6 | % {
$d ="PQRSTU"[($_-1)] + “:”
New-SmbMapping -LocalPath $d -RemotePath \\JOSE-f\Share$_ -Persistent $true
}
# Prepare - Create the test files
1..6 | % {
$f ="PQRSTU"[($_-1)] + “:\testfile.dat”
fsutil file createnew $f (5GB)
fsutil file setvaliddata $f (5GB)
}
# Remove the third cluster node (demo starts with 2 nodes)
Remove-ClusterNode -Cluster JOSE-A -Name JOSE-A3
Start Performance Monitor
Start a performance monitor session
Switch to “Histogram Bar” view to show the performance side-by-side
Add a counter for SMB Server Shares, Data bytes/sec, _total instance for JOSE-A1, JOSE-A2 and JOSE-A3.
Query the cluster shared volume ownership on Cluster A, with 2 nodes
Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize
OwnerNode Name State
--------- ---- -----
JOSE-A1 Cluster Disk 6 Online
JOSE-A1 Cluster Disk 3 Online
JOSE-A1 Cluster Disk 4 Online
JOSE-A2 Cluster Disk 7 Online
JOSE-A2 Cluster Disk 2 Online
JOSE-A2 Cluster Disk 5 Online
Run SQLIO to issue 8KB IOs
C:\sqlio\sqlio.exe -s9999 -T100 -t1 -o16 -b8 -BN -LS -c2000 -frandom -dPQRSTU testfile.dat
![clip_image001[7]](https://msdntnarchive.blob.core.windows.net/media/TNBlogsFS/prod.evol.blogs.technet.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/00/38/13/metablogapi/0172.clip_image001%5B7%5D.png "clip_image001[7]")
Add a 3rd node and wait for it to take 1/3 of the load
Add-ClusterNode -Cluster JOSE-A -Name JOSE-A3
# Wait 2 and a half minutes to transition to the following state.
![clip_image002[9]](https://msdntnarchive.blob.core.windows.net/media/TNBlogsFS/prod.evol.blogs.technet.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/00/38/13/metablogapi/2313.clip_image002%5B9%5D.png "clip_image002[9]")
Re- query the cluster shared volume ownership on Cluster A, now with 3 nodes
Get-ClusterSharedVolume -Cluster JOSE-A | Sort OwnerNode | FT OwnerNode, Name, State -AutoSize
OwnerNode Name State
--------- ---- -----
JOSE-A1 Cluster Disk 3 Online
JOSE-A1 Cluster Disk 6 Online
JOSE-A2 Cluster Disk 5 Online
JOSE-A2 Cluster Disk 7 Online
JOSE-A3 Cluster Disk 2 Online
JOSE-A3 Cluster Disk 4 Online
18) Configuring VMM on JOSE-V
Bring the File Server Cluster under VMM management
Select Fabric and use the option to add Storage Devices
Add a Windows-based file server
![clip_image002[7]](https://msdntnarchive.blob.core.windows.net/media/TNBlogsFS/prod.evol.blogs.technet.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/00/38/13/metablogapi/4885.clip_image002%5B7%5D.png "clip_image002[7]")
Specify the full path to the file server cluster:
![clip_image003[11]](https://msdntnarchive.blob.core.windows.net/media/TNBlogsFS/prod.evol.blogs.technet.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/00/38/13/metablogapi/4478.clip_image003%5B11%5D.png "clip_image003[11]")
Verify the File Server Cluster was properly discovered by VMM
Check the provider
Check the Storage Spaces discovery
Check the Scale-Out File Server and file share discovery
Remove the File Server Cluster node (demo starts with 2 nodes)
Under the properties of the File Server Cluster, remove node 3
Check progress under running Jobs
![clip_image008[6]](https://msdntnarchive.blob.core.windows.net/media/TNBlogsFS/prod.evol.blogs.technet.com/CommunityServer.Blogs.Components.WeblogFiles/00/00/00/38/13/metablogapi/2112.clip_image008%5B6%5D.png "clip_image008[6]")
While running a workload, add a File Server Cluster node
Under the properties of the File Server Cluster, add node 3 specifying the full path of to the server
Check progress under running Jobs
19) Verifying systems’ configuration
# Commands to verify the configuration of all systems
"B1", "B2" | % { Get-VM -ComputerName JOSE-$_ }
"B1", "B2" | % { $_; Get-VM -ComputerName JOSE-$_ | Get-VMHardDiskDrive | FT VMName, ControllerType, ControllerLocation, Path -AutoSize}
Get-SmbOpenFile -CimSession JOSE-A1 | Sort ClientUserName, ShareRelativePath | FT ClientUserName, ShareRelativePath –AutoSize
"X1", "X2" | % { $_; Get-Disk -CimSession JOSE-$_ } | FT -AutoSize
"A", "B", "X" | % { $_; Get-ClusterNode -Cluster JOSE-$_ | FT Cluster, NodeName, State, Id -AutoSize }
"A", "B", "X" | % { $_; Get-ClusterResource -Cluster JOSE-$_ | FT -AutoSize}
20) Issues and FAQs (Frequently asked questions)
Failover Cluster creation in the guest fails.
- Make sure you’re logged on as a domain user, not a local user
- Make sure the Windows Server 2012 R2 integration components are installed in the guest.
- Check for individual warnings and errors in the Failover Clustering validation report.
I can’t add a shared VHDX to a VM. I get a message saying that “the storage where the virtual hard disk is located does not support virtual hard disk sharing.”
- Make sure your using a CSV disk or an SMB Scale-Out file share to store your VHDX files
- Make sure the SMB Scale-Out file server cluster is running Windows Server 2012 R2
21) Final Notes
- Keep in mind that there are dependencies between the services running on each computer.
- To shut them down, start with VMM server, then the Hyper-V servers, then the File Server and finally the DNS/DC, waiting for each one to go down completely before moving to the next one.
- To bring the servers up again, go from the DNS/DC to the File Server Cluster, then the Hyper-V Cluster and finally the VMM Server, waiting for the previous one to be fully up before starting the next one.
- I hope you enjoyed these step-by-step instructions. I strongly encourage you to try them out and perform the entire installation yourself. It’s a good learning experience.
- Let me know how these steps worked for you using the comment section. If you run into any issues or found anything particularly interesting, don’t forget to mention the number of the step.
To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.
Comments
- Anonymous
January 01, 2003
The catch with moving beyond 2 nodes connected to the same JBOD unit is the loss of HBA redundancy on any node beyond the second.2 nodes means 2 HBAs, and thus MPIO, for a full set of redundant connections.Does it really make sense to move into single HBA/Node beyond a two node setup? At least at 6Gbit SAS? - Anonymous
January 01, 2003
The comment has been removed - Anonymous
January 01, 2003
@Yoshihiro Most JBODs have the option to add two modules in the back, each with 2 SAS connections. So they can have 4 SAS ports available for connecting up to 4 computers without the need for SAS expanders. - Anonymous
January 01, 2003
@Mitko I use Net1 primarily to manage the machines. It's a DHCP-enabled network that I use to RDP into the machines. Net1R is my preferred for migration, since it's a 32Gbps RDMA-capable network (everything else in this setup uses 1GbE). - Anonymous
January 01, 2003
@Mitko DimitrovActually, for this one, I used a low-cost IS5022 8-port InfiniBand QDR switch (32Gbps per port). You could also deploy a traditional 10GbE switch with non-RDMA NICs, if that's enough for your workload. - Anonymous
January 01, 2003
Jose,Shouldn't the Hyper-V Role be installed before Failover-Clustering? - Anonymous
January 01, 2003
We utilize a pair of LSI SAS Switches set up with two distinct paths for this purpose. They work very well.Thanks! :) - Anonymous
January 01, 2003
@Philip @JoseDataOn Storage has POC designs of 3 nodes utilizing a single SAS JBOD with redundant paths and HBAs. In all of DataOn Storage POC designs it seems like you can also use the daisy chain SAS ports on the SAS JBOD to connect to the SOFS nodes.www.dataonstorage.com/.../windows-server-2012-storage-spaces-certified-tested-approved-solutions.html - Anonymous
January 01, 2003
@Philip ElderYou are correct. We are trading having 2 SAS paths to the JBOD with MPIO for having 3 or 4 nodes in your cluster. You can argue that you have additional nodes so a single SAS path failure will cause a node failure. There's also the question of performance, since with 2 SAS with MPIO deliver better performance per node. If you want 3 or 4 nodes and dual SAS paths to the JBOD, you need to add SAS expanders to the mix. - Anonymous
January 01, 2003
@YoshihiroJust pick one from the list of certified JBODs for Storage Spaces.See www.windowsservercatalog.com/results.aspx - Anonymous
January 01, 2003
@Yoshihiro The LSI HBA should be fine - Anonymous
January 01, 2003
@Philip As long as you install both, you're fine. You do have to reboot after installing the Hyper-V role. - Anonymous
July 31, 2013
JBOD vender and model , Please.I want to try Failover Scale-Out File Server Cluster in 3 nodes and SAS JBOD. - Anonymous
July 31, 2013
Can I use LSI HBA 9207-8e ?I want to try 3 node cluster with JBOD. - Anonymous
July 31, 2013
What is the propose of Net1 (Corp) ? And from what I understand you use NetR1 (RDMA) for Live migration also ? - Anonymous
July 31, 2013
Thank you Jose.I want to know your HBA and JBOD for this blog.because the list of certified JBODs does not provide to support 3 node. - Anonymous
August 06, 2013
So the big investment is this case is not the storage , but all RDMA cable network equipment - switch / network adapters. Can you share with us , what device you use for RDMA switch ? - Anonymous
March 30, 2014
In this post, I'm providing a reference to the most relevant content related to Windows Server 2012 - Anonymous
June 03, 2014
Pingback from Windows 2012: Storage SMB in alta disponibilità | DevAdmin Blog - Anonymous
July 07, 2014
In this post, I'm providing a reference to the most relevant content related to Windows Server 2012 - Anonymous
November 19, 2014
In this post, I'm sharing my favorite links related to Storage Spaces in Windows Server 2012 R2.