The most important Timer Job for Search you may not know exists

  • Article

What comes to mind when I say "Application Server Administration Services" timer job? Based on the name alone, Search is probably the last answer - it was for me... until I learned just how important this timer job is to Search.

Having failures during a topology change? Go check this timer job. Master merge not being triggered? Go check this timer job. Have a crawl component stuck in starting or failing to provision? Component file shares that don't get created? Web service end points that won't provision? Search Schema that won't propagate to the Search Admin DB or back to the Query Components? UI not allowing you to trigger a backup of the SSA even though you clearly have permissions? Components or SSA in an otherwise unexplained funky state? An SSA that won't Pause or Resume? Crawls stuck on a Crawl Component? Got a ServerID mismatch? Or the Juno Constellation not synching with the Search Admin DB? 

...for all of these, go check that this timer job is running. And sometimes making it more opaque, it has to run on ALL search servers (not just servers with a crawl component) and make sure it's running every minute. This timer job isn't the root cause to all of the issues I've listed above, but it is highly involved in al of these scenarios (I can personally attest to troubleshooting each of these :-) ) and will in most cases provide you many of the missing clues you can't track down. For what it's worth, I generally consider this timer job the real "administration" component (and consider the "Admin Component" shown in the Topology as the Search Index Component).

I've hit enough of these errors enough times that I had to extend the Search Topology script from Knut Brandrud to verify this job is running and uploaded it to the TechNet scripts center:

Get-SPSearchTopologyState script extended to check the synchronization timer job

Just to be clear, this is the exact script provided by Knut Brandrud (link below), but has been extended to include checks for the "Application Server Administration Services Timer Job" status on each Search Server in the Farm. Knut is not associated in any way with this modification, so please blame ...errr, reach out to me if you have any questions about this branch of the script.

His original script can be found at his blog here:

How to view the status of the SharePoint 2013 search service

I hope this saves you some grief!

Comments

  • Anonymous
    July 23, 2015
    Thanks for this script, Brian! It's excellent. I do a lot of PowerShell and I work with Search a lot. One of my customers has two SSAs and so I made some small updates to your script to make it work for multiple SSAs. I'd be happy to send it you if you'd like. Not sure how to do that, so you feel free to email me at brian.laws@summit7systems.<SPAMisEvil>com so that I can send it to you. BTW, your sessions at SPC and Ignite were really, really helpful. Keep up the great work!

  • Anonymous
    August 14, 2015
    Thanks Brian - very much appreciated!! ...and appreciate the offer regarding multiple SSAs. For what it's worth, I generally avoid having multiple SSAs on a single farm and in my opinion, recommend against it if you can avoid it. Being said, it was intentional to write this for a single SSA (e.g. to avoid unintentionally advocating it) Plus, my version of the script was built on top of the one from a member of our PG. I can't speak for him specifically... but I think he may have a similar justification for assuming just one SSA per farm. In either case, I attempted to just keep it in the spirit of what he had already developed. Even though it is supported to do so, there are some quirks to be aware of...  - In ULS, many of the events are completely ambiguous as to which SSA the message relates making troubleshooting and monitoring incredibly difficult (e.g. does the message relate to SSA1 or SSA2?)  - Assuming like topologies across both SSAs, the hardware requirements would effectively double to maintain the same level of performance for each SSA. Being said, other than Central Admin (and a second Configuration DB), there is very little difference in performance between a one giant farm versus two farms with the same number of servers  - Services such as the Search Query & Site Settings ( SQ&SS), the crawler "PerformanceLevel" and service accounts are all farm level configurations       --- A common justification for multiple SSAs is security isolation, but the same Search service account will be used for both SSAs in the same farm       --- This also means there could be implications to restores (In my test farm, I've accidentally trampled the accounts for one SSA when I restored a 2nd SSA and specified different accounts for it...)  - There have been several fixes around multiple SSAs in a single farm... to knowledge, all these have all been fixed and I haven't seen one in a while around this... but why risk it unless you have to. ...there are definitely more quirks, but this gives the gist of my rationale here. I hope this helps, --Brian