Manage storage infrastructure for Azure Stack Hub
This article describes the health and operational status of Azure Stack Hub storage infrastructure resources. These resources include storage drives and volumes. The information in this topic helps you troubleshoot various issues, like when a drive can't be added to a pool.
Volume states
To find out what state volumes are in, use the following PowerShell commands:
$scaleunit_name = (Get-AzsScaleUnit)[0].name
$subsystem_name = (Get-AzsStorageSubSystem -ScaleUnit $scaleunit_name)[0].name
Get-AzsVolume -ScaleUnit $scaleunit_name -StorageSubSystem $subsystem_name | Select-Object VolumeLabel, HealthStatus, OperationalStatus, RepairStatus, Description, Action, TotalCapacityGB, RemainingCapacityGB
Here's an example of output showing a detached volume and a degraded/incomplete volume:
| VolumeLabel | HealthStatus | OperationalStatus |
|---|---|---|
| ObjStore_1 | Unknown | Detached |
| ObjStore_2 | Warning | {Degraded, Incomplete} |
The following sections list the health and operational states:
Volume health state: Healthy
| Operational state | Description |
|---|---|
| OK | The volume is healthy. |
| Suboptimal | Data isn't written evenly across drives. Action: Contact Support to optimize drive usage in the storage pool. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. You may have to restore from backup after the failed connection is restored. |
Volume health state: Warning
When the volume is in a Warning health state, it means that one or more copies of your data are unavailable but Azure Stack Hub can still read at least one copy of your data.
| Operational state | Description |
|---|---|
| In service | Azure Stack Hub is repairing the volume, like after adding or removing a drive. When the repair is complete, the volume should return to the OK health state. Action: Wait for Azure Stack Hub to finish repairing the volume and check the status afterward. |
| Incomplete | The resilience of the volume is reduced because one or more drives failed or are missing. However, the missing drives contain up-to-date copies of your data. Action: Reconnect any missing drives, replace any failed drives, and bring online any servers that are offline. |
| Degraded | The resilience of the volume is reduced because of one or more failed or missing drives as well as outdated copies of data on the drives. Action: Reconnect any missing drives, replace any failed drives, and bring online any servers that are offline. |
Volume health state: Unhealthy
When a volume is in an Unhealthy health state, some or all of the data on the volume is currently inaccessible.
| Operational state | Description |
|---|---|
| No redundancy | The volume has lost data because too many drives failed. Action: Contact Support. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. |
Volume health state: Unknown
The volume can also be in the Unknown health state if the virtual disk has become detached.
| Operational state | Description |
|---|---|
| Detached | A storage device failure occurred which may cause the volume to be inaccessible. Some data may be lost. Action: 1. Check the physical and network connectivity of all storage devices. 2. If all devices are connected correctly, contact Support. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. You may have to restore from backup after the failed connection is restored. |
Drive states
Use the following PowerShell commands to monitor the state of drives:
$scaleunit_name = (Get-AzsScaleUnit)[0].name
$subsystem_name = (Get-AzsStorageSubSystem -ScaleUnit $scaleunit_name)[0].name
Get-AzsDrive -ScaleUnit $scaleunit_name -StorageSubSystem $subsystem_name | Select-Object StorageNode, PhysicalLocation, HealthStatus, OperationalStatus, Description, Action, Usage, CanPool, CannotPoolReason, SerialNumber, Model, MediaType, CapacityGB
The following sections describe the health states a drive can be in:
Drive health state: Healthy
| Operational state | Description |
|---|---|
| OK | The volume is healthy. |
| In service | The drive is doing some internal housekeeping operations. When the action is complete, the drive should return to the OK health state. |
Drive health state: Warning
A drive in the Warning state can read and write data successfully but has an issue.
| Operational state | Description |
|---|---|
| Lost communication | Connectivity has been lost to the drive. Action: Bring all servers back online. If that doesn't fix it, reconnect the drive. If this state persists, replace the drive to ensure full resiliency. |
| Predictive failure | A failure of the drive is predicted to occur soon. Action: Replace the drive as soon as possible to ensure full resiliency. |
| IO error | There was a temporary error accessing the drive. Action: If this state persists, replace the drive to ensure full resiliency. |
| Transient error | There was a temporary error with the drive. This error usually means the drive was unresponsive, but it could also mean that the Storage Spaces Direct protective partition was inappropriately removed from the drive. Action: If this state persists, replace the drive to ensure full resiliency. |
| Abnormal latency | The drive is sometimes unresponsive and is showing signs of failure. Action: If this state persists, replace the drive to ensure full resiliency. |
| Removing from pool | Azure Stack Hub is in the process of removing the drive from its storage pool. Action: Wait for Azure Stack Hub to finish removing the drive, and check the status afterward. If the status remains, contact Support. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. |
| Starting maintenance mode | Azure Stack Hub is in the process of putting the drive in maintenance mode. This state is temporary--the drive should soon be in the In maintenance mode state. Action: Wait for Azure Stack Hub to finish the process and check the status afterward. |
| In maintenance mode | The drive is in maintenance mode, halting reads and writes from the drive. This state usually means Azure Stack Hub administration tasks such as PNU or FRU are operating the drive. But the admin could also place the drive in maintenance mode. Action: Wait for Hub Azure Stack Hub to finish the administration task, and check the status afterward. If the status remains, contact Support. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. |
| Stopping maintenance mode | Azure Stack Hub is in the process of bringing the drive back online. This state is temporary - the drive should soon be in another state, ideally Healthy. Action: Wait for Azure Stack Hub to finish the process and check the status afterward. |
Drive health state: Unhealthy
A drive in the Unhealthy state can't currently be written to or accessed.
| Operational state | Description |
|---|---|
| Split | The drive has become separated from the pool. Action: Replace the drive with a new disk. If you must use this disk, remove the disk from the system, make sure there's no useful data on the disk, erase the disk, and then reseat the disk. |
| Not usable | The physical disk is quarantined because it's not supported by your solution vendor. Only disks that are approved for the solution and have the correct disk firmware are supported. Action: Replace the drive with a disk that has an approved manufacturer and model number for the solution. |
| Stale metadata | The replacement disk was previously used and may contain data from an unknown storage system. The disk is quarantined. Action: Replace the drive with a new disk. If you must use this disk, remove the disk from the system, make sure there's no useful data on the disk, erase the disk, and then reseat the disk. |
| Unrecognized metadata | Unrecognized metadata found on the drive, which usually means that the drive has metadata from a different pool on it. Action: Replace the drive with a new disk. If you must use this disk, remove the disk from the system, make sure there's no useful data on the disk, erase the disk, and then reseat the disk. |
| Failed media | The drive failed and won't be used by Storage Spaces anymore. Action: Replace the drive as soon as possible to ensure full resiliency. |
| Device hardware failure | There was a hardware failure on this drive. Action: Replace the drive as soon as possible to ensure full resiliency. |
| Updating firmware | Azure Stack Hub is updating the firmware on the drive. This state is temporary and usually lasts less than a minute and during which time other drives in the pool handle all reads and writes. Action: Wait for Azure Stack Hub to finish the updating and check the status afterward. |
| Starting | The drive is getting ready for operation. This state should be temporary--once complete, the drive should transition to a different operational state. Action: Wait for Azure Stack Hub to finish the operation and check the status afterward. |
Reasons a drive can't be pooled
Some drives just aren't ready to be in Azure Stack Hub storage pool. You can find out why a drive isn't eligible for pooling by looking at the CannotPoolReason property of a drive. The following table gives a little more detail on each of the reasons.
| Reason | Description |
|---|---|
| Hardware not compliant | The drive isn't in the list of approved storage models specified by using the Health Service. Action: Replace the drive with a new disk. |
| Firmware not compliant | The firmware on the physical drive isn't in the list of approved firmware revisions by using the Health Service. Action: Replace the drive with a new disk. |
| In use by cluster | The drive is currently used by a Failover Cluster. Action: Replace the drive with a new disk. |
| Removable media | The drive is classified as a removable drive. Action: Replace the drive with a new disk. |
| Not healthy | The drive isn't in a healthy state and might need to be replaced. Action: Replace the drive with a new disk. |
| Insufficient capacity | There are partitions taking up the free space on the drive. Action: Replace the drive with a new disk. If you must use this disk, remove the disk from the system, make sure there's no useful data on the disk, erase the disk, and then reseat the disk. |
| Verification in progress | The Health Service is checking to see if the drive or firmware on the drive is approved for use. Action: Wait for Azure Stack Hub to finish the process, and check the status afterward. |
| Verification failed | The Health Service couldn't check to see if the drive or firmware on the drive is approved for use. Action: Contact Support. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. |
| Offline | The drive is offline. Action: Contact Support. Before you do, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. |