Considerations when deploying SharePoint & Office365 Hybrid workloads in a farm utilizing Provider hosted add-ins or workflow manager

  • Article

Updated with Remediation Script and KB Article references - 17th January 2017

This article is written with Thanks and accreditation to Nico Martens who first raised this issue and worked with Microsoft to validate his solution, outlined here . The original post highlighted an issue with Cloud Hybrid Search however this affects all Hybrid workloads utilizing a Server to Server (S2S) trust between SharePoint and Office365.

Scenario: We are going to focus on couple of use cases that can lead to this situation. Intention is to implement Hybrid workload on a new or existing SharePoint Farm with either

  1. You have a SharePoint 2013 or SharePoint 2016 farm where you plan to implement provider hosted add-ins and/or associate with workflow manager.
  2. You have a SharePoint 2013 or SharePoint 2016 farm that already has provider hosted add-ins and/or leverages workflow manager.

When you try to establish a S2S trust via the Cloud SSA onboarding script or Hybrid picker, the authentication realm of the on premises Farm is updated to match the Office 365 tenant context id. Within the script we set it using Set-SPAuthenticationRealm. Once the authentication realm is changed, existing SharePoint Add-ins fail to authenticate; users will get a HTTP 401 when they are redirected to the add-ins.

The reason for this authentication failure is due to how provider hosted Add-ins are authorized to access SharePoint. SharePoint Add-ins are associated with SPTrustedSecurityTokenIssuers using the IssuerId and on request it tries to get a token from the Secure Token issuer (STS). The token issuers are tied to the authentication realm and once the realm is changed, the SharePoint Add-ins can no longer authenticate successfully. The trustedtokenissuer with the correct IssuerId still exists in the farm but it is associated with the previous authentication realm.

In effect; the RegisteredIssuerName is IssuerId@OldAuthRealmGuid where the oldAuthRealmGuid no longer matches the current AuthRealmGuid. The Add-in fails to authenticate because no matching tokenissuer can be found by the STS. The error message can be seen in ULS and clearly indicate that the token issuer is no longer trusted (because its RealmID doesn’t match the farm anymore)  SPApplicationAuthenticationModule: Failed to authenticate request, unknown error. Exception details: System.IdenitytModel.Tokens.SecurityTokenException: The issues of the token is not a trusted issuer.

So, based on our two scenarios listed above, what can we do? Well, the answer depends on which scenario you are trying to solve. Let’s look at them in turn.

Scenario One

Here you have not yet implemented Hybrid workloads, nor have you implemented Add-ins or workflow manager. In this scenario, the only thing you need to do is ensure you implement the hybrid workloads requiring S2S, before you implement the Provider hosted Add-ins or workflow manager. The Add-ins will then be registered after the SPAuthenticationRealm has been updated to match the Office 365 tenant contextID and as such will always work because the Realm will not change again. Even if additional hybrid workloads are added or reconfigured, the RealmID will remain the same as the Office 365 tenant context ID.

Scenario Two

Here you have an existing set of Add-ins already deployed and these have of course been registered using the RegisteredIssuerName containing the original RealmID of the farm. When you implement Hybrid, the RealmID is changed to match the tenant ID of your Office365 subscription causing add-ins to cease working as explained above. To restore SharePoint add-in functionality, we have to re-register the provider hosted add-ins but using the RegisteredIssuerName containing the new RealmID. After this, you need to reapply the Add-in permissions for each individual Add-in instance, following which the Add-ins will operate successfully once more.

This behavior is by design. When implementing SharePoint Hybrid features you need to carefully consider the additional workloads at play in the farm especially when provider hosted add-ins are being used.

The second aspect of Scenario 2 involves workflow manager.

The Workflow Manager TrustedSecurityTokenIssuer uses a wildcard realm (e.g. “<guid>@*”) instead of a specific realm identifier. This means workflows will keep working even  after changing the SPAuthenticationRealm.

Remediation process

The authors have developed a solution to repair the PHA instances on a SharePoint Server farm which is based on (with acknowledgement and permission) the sterling work done by Nico Martens on his post.

Additionally, if you deploy the hybrid features after January 8th 2017, this remediation logic is now also built into the hybrid picker from Office 365.  For hybrid search (outbound/inbound query federation or Cloud Hybrid Search Service Application) a manual approach is needed to remediate this scenario. Microsoft have released a KB article with the relevant information here and we have now developed a pre-built remediation scrip based of this information for hybrid search based scenarios.

The remediation script is located here. Download and execute on your farm to remediate the problems with PHAs

For Workflow Manager connection the process is slightly different and involves recreation of the WorkFlow Manager Proxy Connection. The Script to perform this action is here. Download and execute it on the farm to repair the connection.

Workflow manager fix up script along with Cloud Search Service Application and Supporting Scripts for Deployment and Configuration can be downloaded here.