SharePoint 2013: Remove User Profile Service Application process is stuck

Problem

Attempted to remove a user profile service application from a small SharePoint Server 2013 farm using routine PowerShell commands.  First attempted to remove the service application's proxy:

$proxy = Get-SPServiceApplicationProxy | ?{$_.ID -eq '[proxy ID]'}
Remove-SPServiceApplicationProxy $proxy -confirm:$False

This completed within several minutes, and the proxy was no longer listed in the Manage service applications page in the farm's Central Administration  Then attempted to remove the service application itself:

$spapp = Get-SPServiceApplication | ?{$_.ID -eq '[service application ID]'}
Remove-SPServiceApplication $spapp -confirm:$False

This did not complete on the day it was run. No error was displayed in the shell.

Later that day, observed that the user profile service application status was presented in Central Administration (on the Manage service applications page) as Stopping.

tbd

Troubleshooting

01) Restart SharePoint Timer Service

Two days after initiating the service application removal, launched the Windows server Services applet, stopped and started the SharePoint Timer Service a couple of times, and then waited awhile, but nothing changed.

02) Checked ULS logs. Found the start of the process followed eventually by these:

SPTimerJobAsyncResult '496289bd-d0f7-4779-84d5-582caa060ebc': Timer job has not completed.  Blocking until job completes.
Blocking until timer job '496289bd-d0f7-4779-84d5-582caa060ebc' has completed.

       and no more messages associated with that correlation ID over the next few hours.

03) Opened an elevated DOS shell, and then executed an STSADM command:

stsadm -o provisionservice -action stop -servicetype "Microsoft.Office.Server.Administration.UserProfileService, Microsoft.Office.Server.UserProfiles, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" -servicename "User Profile Service"

       Waited awhile, but no changes were observed: the service application status continued to be Stopping.

04) Reboot server

Rebooted the server (simple one SharePoint server, one SQL Server farm).  Checked and found status still Stopping.

05) Unprovisioned the service application:

$spapp = Get-SPServiceApplication | ?{$_.ID -eq '[service application ID]'}
$spapp.Unprovision()

      After several minutes, the status of the service application as seen through Central Administration Manage service applications page changed to Stopped.

06) Checked the status of the service application:

get-spserviceapplication | ft DisplayName,ID,Status -auto

      Found the status Disabled.

07) Attempted again to remove the service application using elevated PowerShell:

$spapp = Get-SPServiceApplication | ?{$_.ID -eq '[service application ID]'}
Remove-SPServiceApplication $spapp -confirm:$False

      Refreshing the Manage service applications page showed the status had changed to Stopping.  Prompt still hadn't returned in shell.

08) Checked ULS logs

SPTimerJobAsyncResult '44456fca-4755-435a-859d-fe2314f2161b': Timer job has not completed.  Blocking until job completes.
Blocking until timer job '44456fca-4755-435a-859d-fe2314f2161b' has completed.

       No more messages associated with this correlation ID appeared after waiting several minutes.

09) Restarted SharePoint Timer Service

Stopped and started SharePoint Timer Service a couple of times and then waited a few more minutes.

10) Check for prompt in shell

       Prompt returned, indicating completion of command.

11) Refreshed the Manage service applications page.

User Profile Service application no longer listed.

12) Check ULS logs. Found this entry associated with correlation ID 13 minutes after the last one:

The configuration database was queried for a non-existent object with the id 44456fca-4755-435a-859d-fe2314f2161b. Most
commonly, this is caused by removing an SPFeatureDefinition without writing upgrade code to remove references to the 
feature from each site.

       Followed by a few more messages and then completion of the process quickly thereafter.

Solution

  1. Terminate the Remove-SPServiceApplication job
  2. Unprovision the service application
  3. Run Remove-SPServiceApplication again

References

Notes

  • The remove process may well have completed on the first attempt had more time been given for the process to complete.  However, it wasn't clear that this would happen after waiting two days.
  • It's not clear that stopping and starting the SharePoint Timer Service helped accelerate completion of the process, but it didn't seem to hurt the process.  That's a factor that I haven't yet isolated in this solution.