SharePoint 2013 Hybrid Search Setup issues

Reference SharePoint 2013 environment has 

  • 2 App servers, 
  • 2 WFE servers, 
  • SQL Server with Always On Availability Groups. 
  • 1 App server has Central Admin and other service applications including on-prem Search SA, and 
  • the other App server was pretty much free, so I thought to setup the Cloud Search on the 2nd App server for load balancing purpose.

On first attempt of running CreateCloudSSA.ps1, I got an error "Failed to call GetTypes on assembly Microsoft.AnalysisServices.SPAddin, Version=12.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxx. Could not load file or assembly 'Microsoft.AnalysisServices.SPClient, Version=12.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxx or one of its dependencies. The system cannot find the file specified."

On further troubleshooting about this error, I saw that the other servers of my SP 2013 Farm had SQL Server 2014 PowerPivot component installed, but this App server didn't have it and that's why it was causing this issue. So I downloaded the PowerPivot from Microsoft website and installed it on this App server followed by running the SharePoint Config Wizard.

Then again I tried to run the CreateCloudSSA.ps1, I hit another issue where the Cloud Search Service Application was created, all 4 databases were created, but topology was not getting activated. I was getting this error "Topology activation failed. No system manager locations set, search application might not be ready yet."

I searched a lot on Internet about this error, but didn't really get any fruitful solution. Finally I set the Verbose logging for SharePoint Search in Central Admin and tried to activate the topology. Then I saw below errors in ULS log under NodeRunner process:

Error#1

Unexpected exception in node activator: Microsoft.Ceres.CoreServices.Services.Configuration.ConfigException: Lease Core Service not loaded. Is the DLL missing? ---> System.TypeLoadException: No matching type registered for: Microsoft.Ceres.SharePoint.Lease.SharePointLeaseComponent. The library with the component(or type) is most likely missing.    

 at Microsoft.Ceres.CoreServices.Node.TypeBinding.TypeRegistry.GetRegisteredType(String name)    

 at Microsoft.Ceres.CoreServices.Node.NodeActivator.InitializeCoreComponents(NodeConfigurationStorage store, INodeConfiguration nodeConfig, ITypeRegistry typeRegistry)     -

 -- End of inner exception stack trace ---    

 at Microsoft.Ceres.CoreServices.Node.NodeActivator.InitializeCoreComponents(NodeConfigurationStorage store, INodeConfiguration nodeConfig, ITypeRegistry typeRegistry)    

 at Microsoft.Ceres.CoreServices.Node.NodeActivator.ActivateNode(IDictionary`2 configuration)

Error#2

AssemblyReflector : Failed to load dependency assembly in load context and in load from context. Assembly: BihConsumerInterop System.IO.FileNotFoundException: Unable to find the specified file.

Error#3

AssemblyReflector : Failed to load dependency assembly in load context and in load from context. Assembly: System.ServiceModel System.IO.FileNotFoundException: Unable to find the specified file.

On further checking about these errors, I found that the .NET Framework 3.5 Feature wasn't installed on the App server. After installing that, I rebooted the server and tried again and it worked!