Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)

Applies to: FAST Search Server 2010

If the farm deployment includes a backup indexer row, you must manually change the deployment configuration if a non-recoverable server error occurs on the primary indexer server. You must also synchronize the indexer servers after you have replaced the failed server.

Follow these steps to replace a primary indexer server if a non-recoverable hardware failure has occurred:

• Step 1: Pause item processing and crawling

• Step 2: Reconfigure the backup indexer server to become the new primary indexer server

• Step 3: Install and configure a new backup indexer server

• Step 4: Resume item processing and crawling

Step 1: Pause item processing and crawling

You must pause item processing and crawling before you reconfigure the indexer servers.

To pause item processing and crawling

1. Pause crawling:

   • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about pausing a crawl.

   • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about suspending a crawl.

   • If you use another FAST Search specific connector, make sure that all crawling is paused or stopped.

2. Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

   1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

   2. Inspect the Web Analyzer schedule status:

      waadmin ShowStatus
      

      Note

      Refer to waadmin.exe reference if you use multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status. If the Schedule status is set to paused, perform the following steps:

      1. Set the view to scheduled state:

         waadmin enqueueview
         

      2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and verify that the Schedule status is set to running.

   3. Stop all Web Analyzer processing:

      waadmin AbortProcessing
      spreladmin AbortProcessing
      

3. Wait until all item processing is finished. On the administration server, type the following command:

   psctrl status
   

   Note

   All processor servers must report "idle" before you proceed to the next step.

Step 2: Reconfigure the backup indexer server to become the new primary indexer server

Follow these steps to ensure that the new primary indexer server contains the same set of indexed items as the failed primary indexer server.

To reconfigure the backup indexer server to become the new primary indexer server

1. Ensure that the failed primary indexer server is shut down or disconnected.

2. Modify the deployment configuration (deployment.xml).

   1. On the administration server, open the deployment configuration file: <FASTSearchFolder>\etc\config_data\deployment\deployment.xml.

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

      Locate the host element that specifies the failed primary indexer server. The column attribute of the searchengine element indicates the index column for this indexer.

      The following example shows an extract of a deployment.xml file with the host definition for the primary and backup indexer for index column 0. The first host element (fs4sp2.contoso.com) specifies the primary indexer server. This is the server that is not working and must be replaced. The second host element (fs4sp5.contoso.com) specifies the backup indexer server. The searchcluster element specifies which server that is the primary indexer and which server that is the backup indexer, by using the row element. Typically, row 0 corresponds to the primary indexer server.

      <host name="fs4sp2.contoso.com">
          <content-distributor />
          <searchengine row="0" column="0" />
          <document-processor processes="12" />
        </host>
        ...
        <host name="fs4sp5.contoso.com">
          <query />
          <searchengine row="1" column="0" />
        </host>
        ...
        <searchcluster>
            <row id="0" index="primary" search="true" />
            <row id="1" index="secondary" search="true" />
        </searchcluster>
      

   2. Change the host element for the two indexer servers. The current backup indexer server will now take the role as primary indexer server. The following example shows the new configuration, where the new primary indexer server is the host fs4sp5.contoso.com:

      <host name="fs4sp5.contoso.com">
          <content-distributor />
          <searchengine row="0" column="0" />
          <document-processor processes="12" />
        </host>
        ...
        <host name="fs4sp2.contoso.com">
          <query />
          <searchengine row="1" column="0" />
        </host>
        ...
        <searchcluster>
            <row id="0" index="primary" search="true" />
            <row id="1" index="secondary" search="true" />
        </searchcluster>
      

      Important

      In the deployment example, we have switched the host names for the two host elements, and that way changed the content-distributor and query roles for the two servers. This is recommended in order to keep the same distribution of server roles after the switch of primary indexer server. This means that you must update the connection information for the Query SSA and Content SSA, which is covered later in this procedure.
      It is also possible to switch the row numbers for the two servers, and keep the other server roles assigned to the same servers. This ensures that there will be no change to the SSA connection information, but the performance characteristics of the servers may change.

3. Reconfigure the deployment to switch the primary and backup indexer roles for this index column:

   1. Apply the new configuration to the administration server. Perform the following steps on the administration server:

      1. Stop the FAST Search for SharePoint service.

      2. Stop the FAST Search for SharePoint Monitoring service.

      3. To recreate configuration files, in a Microsoft FAST Search Server 2010 for SharePoint shell, type the following command:

         Set-FASTSearchConfiguration
         

      4. Start the FAST Search for SharePoint service.

         Note

         This will also start the FAST Search for SharePoint Monitoring service.

      5. To verify that all processes are running, type the following command:

         nctrl status
         

         When all processes are running OK, proceed to the next step.

   2. Apply the new configuration to the other servers in the FAST Search Server 2010 for SharePoint farm, except the failed server. Perform the following steps on one server at a time:

      1. If the server has the query matching or indexer role (searchengine element in deployment.xml), delete the following file:

         <FASTSearchFolder>\etc\rtsplatformcache.xml

      2. Stop the FAST Search for SharePoint service.

      3. Stop the FAST Search for SharePoint Monitoring service.

      4. To recreate configuration files, in a Microsoft FAST Search Server 2010 for SharePoint shell, type the following command:

         Set-FASTSearchConfiguration
         

      5. Start the FAST Search for SharePoint service.

      6. To verify that all processes are running, type the following command:

         nctrl status
         

4. If you have moved content-distributor or query roles to new servers in deployment.xml, you must also perform the steps in Make the necessary changes on Microsoft SharePoint Server.

   Important

   For the deployment example earlier in this article, you must perform this step. The content-distributor and query roles are changed for the two servers. This is done to keep the same distribution of components across the servers in the farm.

5. Verify that the new primary indexer server is properly deployed. At the administration server, type the following command:

   indexerinfo --row=<rowNumber> --column=<columnNumber> status
   

   <rowNumber> is the row number for the primary indexer row, typically 0.

   <columnNumber> is the index column number.

   Verify that the following status information is correct:

   • indexer element: status="running ok"

   • indexer/column_role element: state="master" backups="0"

     state="master" indicates that this is the new primary indexer server for this index column.

     backups="0" indicates that there is not yet any working backup indexer for this index column.

6. Determine if you want to replace the failed server at this stage or later.

   • If you want to configure a new backup indexer server at this stage, proceed with Step 3: Install and configure a new backup indexer server.

   • If you do not want to configure a new backup indexer server at this stage, proceed with Step 4: Resume processing and crawling. You must then follow the steps in Replace a failed backup indexer server at a later stage.

     Important

     Until you have configured a new backup indexer server, your deployment will not be fault-tolerant for all components. In the deployment example earlier in this article, index column 0 will not have a backup indexer. The query processing role (query role in deployment.xml) is now associated with the backup indexer server that is not working. Queries will be served correctly as long as you have at least one other server in your farm with the query processing role.

Step 3: Install and configure a new backup indexer server

Follow these steps to create a new backup indexer server. You have already configured the backup indexer server to become the new primary indexer server.

To install and configure a new backup indexer server

1. Verify that the host name for the new server is correct in the deployment configuration file (deployment.xml).

2. Stop the new primary indexer server you configured in Step 2: Reconfigure the backup indexer server to become the new primary indexer server. On the primary indexer server, type the following command:

   nctrl stop indexer
   

3. Install the new server and reconfigure the deployment to add a backup indexer:

   Important

   When you perform this step, do not restart the new server after you have installed and configured it. You restart the server later in this procedure.

   Follow the steps in Reconfigure the deployment including adding new servers, but do not restart the new server.

4. If you have changed the host name of the new server, update the connection information in the SharePoint Server 2010 farm. Perform the steps in Make the necessary changes on Microsoft SharePoint Server.

5. Copy all FIXML files from the primary indexer server to the new backup indexer server.

   The indexer uses the FIXML files as input to the indexing process. The FIXML files are stored in the following directory on the indexer servers:

   <FASTSearchFolder>\data\data_fixml\

   Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

   1. On the new backup indexer server, create two folders for the index data that you will copy. At the Windows command prompt, type the following commands:

      mkdir <FASTSearchFolder>\data
      mkdir <FASTSearchFolder>\data\data_fixml
      mkdir <FASTSearchFolder>\data\ftStorage
      

   2. On the new backup indexer server, at the Windows command prompt, copy the data_fixml folder from the source server:

      robocopy /E /MT:100 /NFL /COPYALL /LOG:\incoming\robocopy_fixml.log <source_path>\data\data_fixml <FASTSearchFolder>\data\data_fixml
      copy <source_path>\data\ftStorage\processed_checkpoint.txt <FASTSearchFolder>\data\ftStorage
      

      Where <source_path> is the network path of the FAST Search Server 2010 for SharePoint installation folder on the primary indexer server.

      Note

      The /LOG: folder (\incoming) does not exist by default and you must have write access to the folder to run the command.

6. Restart the new backup indexer server.

7. Start the primary indexer server. On the primary indexer server in the affected index column, type the following command:

   nctrl start indexer
   

8. Verify that the new backup indexer server is registered properly with the primary indexer server. At the administration server, type the following command:

   indexerinfo --row=<rowNumber> --column=<columnNumber> status
   

   <rowNumber> is the row number for the primary indexer row, typically 0.

   <columnNumber> is the index column number.

   Verify that the following status information is correct:

   • indexer element: status="running ok"

   • indexer/column_role element: state="master" backups="1"

     backups="1" indicates that the primary indexer server has identified the new backup indexer server.

Step 4: Resume item processing and crawling

Follow these steps to resume item processing and start a full crawl.

To resume item processing and crawling

1. Resume processing for the Web Analyzer components. On the administration server, type the following commands:

   waadmin EnqueueView
   spreladmin Enqueue
   

2. Start a full crawl:

   • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.

   • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.

   • If you use another FAST Search specific connector, make sure that a full new crawl is initiated.

   Note

   We recommend to start a new full crawl, as there is a risk that items may be lost at the time of the primary indexer server failure. This will include items that are received correctly by the old primary indexer server just before the failure, but that have not yet been copied to the backup indexer server.

See Also

Concepts

Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)
Add a backup indexer row (FAST Search Server 2010 for SharePoint)
Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)

Other Resources

About FiXML files