SCSM Troubleshooting: Workflow Intermittent Failures due to Too Many Failed Workflow Instances
If you are having an issue with email notifications sometimes not triggering in SCSM on every ticket as well as Change Requests not completing even after 15 minutes when the last activity has been finished and marked as such, read on!
The first thing to check is the speed of returning your workflow instances. Navigate to [SCSM Console - Administration - Workflows - Status] and see how slow it is to load the SCSM tasks. Ex. click on a workflow and clicked "All Instances", if it took +5 minutes to refresh then you have an issue!
While the issue shows it is simply caused from too many failed workflows being present, the actual damage being done is the fact that due to too many failed workflows, new workflows weren't even being created let alone running when they should've been.
Long story short, if notifications from SCSM seems to be intermittently failing, you may have too many failed workflow instances present and will have to mark them as ignored so new workflows will be able to get created due to SCSM "working" as designed, not as intended.
The following will require basic C# coding skills using a tool such as Visual Studio 2015 Express Edition.
Create a new console project and use the SCSM C# SDK (copied into project and referenced Microsoft.EnterpriseManagement.Core.dll from an SCSM Server: C:\Program Files\Microsoft System Center 2012 R2\Service Manager\SDK Binaries ) to retry failed notification subscriptions:
//References to add with other references after referencing SCSM C# SDK Microsoft.EnterpriseManagement.Core.dll
using Microsoft.EnterpriseManagement;
using Microsoft.EnterpriseManagement.Subscriptions;
//Function to Retry Notification Workflow Instances
private void RetryNotificationWorkflowInstances()
{
EnterpriseManagementGroup emg = new EnterpriseManagementGroup("SCSMSERVER");
//MONSTER WORKFLOW LIST
List<IWorkflowSubscriptionBase> wfSubs = emg.Subscription.GetSubscriptionsByCriteria(new ManagementPackRuleCriteria("Name LIKE '%'")).ToList();
foreach (IWorkflowSubscriptionBase wfSub in wfSubs)
{
if (wfSub is NotificationSubscription)
{
IList<SubscriptionJobStatus> wfSubFailedInstances = emg.Subscription.GetFailedSubscriptionStatusById(wfSub.Id.Value);
foreach (SubscriptionJobStatus wfSubFailedInstance in wfSubFailedInstances)
{
//Retry
emg.Subscription.RetryFailedSubscription(wfSub, wfSubFailedInstance);
}
}
}
}
The above should execute and execute rather quickly, 5 to 10 seconds.
Then wait 15 minutes before doing anything else to allow SCSM to clean itself in the background.
It is very possible that due to such slow performance no notifications were sent on the retry. This is due to SCSM design where too many failed workflows end up slowing down the system enough to have it not create new workflows instances to run.
So the next and final step is to Ignore ALL failed workflow instances!
//References to add with other references after referencing SCSM C# SDK Microsoft.EnterpriseManagement.Core.dll
using Microsoft.EnterpriseManagement;
using Microsoft.EnterpriseManagement.Subscriptions;
//Function to Ignore ALL FAILED Workflow Instances
private void IgnoreALLFAILEDWorkflowInstances()
{
EnterpriseManagementGroup emg = new EnterpriseManagementGroup("SCSMSERVER");
//MONSTER WORKFLOW LIST
List<IWorkflowSubscriptionBase> wfSubs = emg.Subscription.GetSubscriptionsByCriteria(new ManagementPackRuleCriteria("Name LIKE '%'")).ToList();
foreach (IWorkflowSubscriptionBase wfSub in wfSubs)
{
IList<SubscriptionJobStatus> wfSubFailedInstances = emg.Subscription.GetFailedSubscriptionStatusById(wfSub.Id.Value);
foreach (SubscriptionJobStatus wfSubFailedInstance in wfSubFailedInstances)
{
//Ignore
emg.Subscription.IgnoreFailedSubscription(wfSubFailedInstance);
}
}
}
If you have many workflows, expect this to take over 5 minutes. Personally, running this took around 6 minutes the first time (running it after that took about 10 seconds).
Now you should be able to confirm that the SCSM Console Workflow Instances are loading far quicker. Also, confirm if any Change Requests that weren't completed when all activities were finished are now being completed.