OpsMgr: Monitoring Closed Monitor Alerts with a Powershell-based Monitor

"HEY!!    You have monitors in a degraded state but their alerts MAY have been closed incorrectly ! If these monitors are not reset back to Healthy, the same condition that generated their alerts may happen again but no alerts will be generated because the health states have not changed."

This solution uses the Powershell-based monitor type and wizard featured in my first post (can now be downloaded from the TechNet Gallery), for self-monitoring - to find and alert on monitor alerts that may have been incorrectly closed.

For this Powershell-based unit monitor, the following values were used in General Properties.
Note: This monitor targets the RMS Emulator class, and hence ONLY one instance of the workflows will run on the Management Server assigned with the RMS Emulator role.

        l

The polling frequency should be set to 1 day or more and to run during non-business hours as the Powershell script used will search ALL alerts still available in the Operations Console including their owner (source) and can affect the overall performance of the RMS emulator if there is a large number of alerts and managed objects to search through.

The Powershell script used by this monitor to find and alert on monitor alerts that may have been incorrectly closed, has been modified to return its output in a Propertybag.

Notice that the ”import-module operationsmanager” and the ”New-SCOMManagementGroupConnection” cmdlets were used to load the OperationsManager module and connect to the management group.

Here is the script used:

This script searches for closed Monitor Alerts that are not resolved by “System” and alerts if their corresponding monitor are still in Warning or Critical state:

import-module operationsmanager
New-SCOMManagementGroupConnection

$FullList=""
$newline = "`r`n"

# SCOM PropertyBag
$API = new-object -comObject "MOM.ScriptAPI"
$PropertyBag = $API.CreatePropertyBag()

$monitorAlerts = get-SCOMalert | where-object {$_.IsMonitorAlert -eq "True" -and $_.ResolutionState -eq "255" -and $_.ResolvedBy -ne "System"}
foreach ($monitorAlert in $monitorAlerts)
{
$locatedmonitor = get-SCOMmonitor | where-object {$_.ID -eq $monitorAlert.MonitoringRuleID}
$targettedclass = get-SCOMClass | where-object {$_.ID -eq $monitorAlert.MonitoringClassId}

$monitoredobject = $targettedclass | get-SCOMmonitoringObject | where-object {$_.ID -eq $monitorAlert.MonitoringObjectId}

$monitorslist = $null
$monitorslist = new-object "System.Collections.Generic.List[Microsoft.EnterpriseManagement.Configuration.ManagementPackMonitor]"
$locatedmonitor | foreach{ $monitorslist.add($_)}

if($monitoredobject.GetMonitoringStates($monitorslist) | where-object { $_.HealthState -eq "Warning" -or $_.HealthState -eq "Error"})
{
$output1 = $monitoredobject.GetMonitoringStates($monitorslist) | select MonitorDisplayName
$output1 = $output1 -replace "@{MonitorDisplayName=", ""
$output1 = $output1 -replace "}", ""
$output2 = $monitoredobject.FullName

$FullList = $FullList + "Monitor: " + $output1 + " Targetting: " + $output2 + $newline
}
}

if($FullList -eq "")
{
$PropertyBag.AddValue("State","OK")
$PropertyBag.AddValue("Description", "You are good !")
}
else
{
$PropertyBag.AddValue("State","ERROR")
$outputLongLine = "HEY!! You have monitors in a degraded state but their alerts MAY have been closed incorrectly ! They are: " + $newline + $FullList + "Please Note: " + $newline + "If these monitors are not reset back to Healthy, the same condition that generated their alerts may happen again but no alerts will be generated because the health states have not changed."

$PropertyBag.AddValue("Description", $outputLongLine)
}

$PropertyBag

Building expressions based on the value in the Propertybag and mapping monitor conditions to health states were very straight forward as follows. The Expression Builder Pages builds expression that looks for a particular value from the Propertybag that the data source outputs (Property[@Name='State'] ).

The name of the value in the Propertybag was specified in the alert context variable: $Data/Context/Property[@Name='Description']$

To force an alert, close any monitor alerts in the Active Alerts view. Here is an example of the active alert generated and the state change recorded:

Attached with this post are both the self-monitoring - monitor alerts incorrectly closed management pack (TakeAWei.SM.Monitor.Alerts.Incorrectly.ClosedMP.xml) and the sealed library management pack being referenced. Both management packs will need to be imported together via the Operations Console.

Disclaimer:  

All information on this blog is provided on an as-is basis with no warranties and for informational purposes only. Use at your own risk. The opinions and views expressed in this blog are those of the author and do not necessarily state or reflect those of my employer.

MonitorAlertIncorrectlyClosed.zip

Comments

• Anonymous
  December 13, 2013
  Does this require modification to run on scom 2007 R2

• Anonymous
  January 16, 2014
  @Rm031470, yes, the PowerShell script used will require some minor modifications to be OpsMgr 2007 R2 compatible.

• Anonymous
  August 27, 2015
  Is it possible to convert this to 2007R2? I have no clue how to do this. Seems like a very useful script!

• Anonymous
  October 11, 2015
  Great idea... have noticed that once you reset it still alerts on the old one (presumably until it's groomed out) - any thoughts on a way round this?  

• Anonymous
  October 11, 2015
  @scommer, a quick work around would be to right-click and close the alert again and make sure its resolvedby another user so the it would get ignored by the get-SCOMalert cmdlet after its being addressed, like $monitorAlerts = get-SCOMalert | where-object {$.IsMonitorAlert -eq "True" -and $.ResolutionState -eq "255" -and $.ResolvedBy -ne  "System" -and $.ResolvedBy -ne <<another user>>}

• Anonymous
  October 12, 2015
  Thanks for the quick reply Wei... will try that.  thanks :)