Configuring Optional Item Processing

Applies to: SharePoint Server 2010

In this article
Customizing optionalprocessing.xml
File Format for optionalprocessing.xml
Property Extraction
Document Conversion
Offensive Content Filtering
Metadata Extraction

This article describes how to update the configuration file for the optional item processing stages in the pipeline, as follows:

• Customizing optionalprocessing.xml

• File Format for optionalprocessing.xml

• Property Extraction

• Document Conversion

• Offensive Content Filtering

• Metadata Extraction

Customizing optionalprocessing.xml

You enable or disable the optional item processing stages in the optionalprocessing.xml configuration file.

This configuration file is read every time that the item processors are reset, started, or restarted. The file must contain the name and activation status for each optional stage. By default, all optional processing stages are deactivated.

To modify this configuration file, you must be a member of the FASTSearchAdministrators local group on the FAST Search Server 2010 for SharePoint administration server.

Use a text editor or XML editor to change this file.

To change the optionalprocessing.xml file

1. On the FAST Search Server 2010 for SharePoint administration server, edit <FASTSearchFolder>\etc\config_data\DocumentProcessor\OptionalProcessing.xml.

   Where <FASTSearchFolder> is the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

2. On the FAST Search Server 2010 for SharePoint administration server, run the following command.

   <FASTSearchFolder>\bin\psctrl reset

   This will reset all currently running item processors in the system.

File Format for optionalprocessing.xml

The optionalprocessing.xml configuration file has the following syntax.

<optionalprocessing>
  <processor name="personnameextraction" active="yes|no" />
  <processor name="XMLMapper" active="yes|no" />
  <processor name="OffensiveContentFilter" active="yes|no" />
  <processor name="FFDDumper" active="yes|no" />
  <processor name="wholewordsextractor1" active="yes|no" />
  <processor name="wholewordsextractor2" active="yes|no" />
  <processor name="wholewordsextractor3" active="yes|no" />
  <processor name="wordpartextractor1" active="yes|no" />
  <processor name="wordpartextractor2" active="yes|no" />
  <processor name="MetadataExtraction" active="yes|no" />
  <processor name="SearchExportConverter" active="yes|no" />
</optionalprocessing>

Table 1 describes the optional item processing stages.

Table 1. Optional item processing stages

The following example shows how to enable the generation of a personnames crawled property that contains people names extracted from the processed content. You enable the stage by changing the value of the active attribute to yes.

<optionalprocessing>
    <processor name="personnameextraction" active="yes"/>
</optionalprocessing>

The following example shows how to enable mapping of XML content to crawled properties.

<optionalprocessing>
    <processor name="XMLMapper" active="yes"/>
</optionalprocessing>

Property Extraction

Property extraction is a process that extracts information from the visible textual content of an item and stores that information as additional crawled properties for the document.

There are three built-in property extraction stages in the FAST Search Server 2010 for SharePoint item processing pipeline, which do the following:

• The person name extractor extracts names of persons, based on a generic dictionary. By default, this stage is disabled, as FAST Search Server 2010 for SharePoint includes other features related to person name extraction (author property and people search feature). If you also want to extract names that are not specific to your company or organization, you can enable the stage in optionalprocessing.xml.

• The location extractor extracts names of geographical locations, based on a generic dictionary. By default, this stage is enabled. If this property extraction is not relevant for your application, you do not have to map the resulting crawled property to a managed property in the index.

• The company extractor extracts names of companies, based on a generic dictionary. By default, this stage is enabled. If this property extraction is not relevant for your application, you do not have to map the resulting crawled property to a managed property in the index.

The built-in property extraction stages support the following languages:

• Arabic

• Dutch

• English

• French

• German

• Italian

• Japanese

• Norwegian

• Portuguese

• Russian

• Spanish

The default dictionaries for people, location, and company, have been created in order to achieve a reasonable coverage for public news content in the languages indicated earlier in this section.

You can modify the built-in property extractors by adding inclusion lists and exclusion lists. For information, see Manage Property Extraction (FAST Search Server 2010 for SharePoint) on Microsoft TechNet.

You can add custom property extractors to the pipeline. For information, see Creating a Custom Property Extractor.

Document Conversion

The processing stage named SearchExportConverter controls the FAST Search Server 2010 for SharePoint Advanced Filter Pack. This feature enables text and metadata extraction from several hundred file formats, complementing the document formats that are supported by the standard Filter Pack. By default, the Advanced Filter Pack is disabled.

You can also deploy custom IFilter components that are developed for specific file formats. This is controlled via the user_converter_rules.xml configuration file. For information, see Configure FAST Search Server for SharePoint to use a Third-Party IFilter.

Offensive Content Filtering

The FAST Search Server 2010 for SharePoint offensive content filtering is implemented as a separate item processing stage. Item content that is run through the filter is compared to predefined terms in dictionaries. The output of the filter is an overall score that indicates the likeliness that an item is pornographic. The item's offensive score is written to the crawled property OCF::Score. Any item that exceeds the score threshold of 30 will be dropped from indexing.

The FAST Search Server 2010 for SharePoint offensive content filter uses single words and multiword expressions as the basis for the filtering.

By default, the offensive content filter is not enabled. You enable it by using the activation key OffensiveContentFilter in optionalprocessing.xml, as shown in the following example.

<optionalprocessing>
      <processor name="OffensiveContentFilter" active="yes"/>
</optionalprocessing>

Offensive content filtering supports following languages:

• Arabic

• Chinese

• Czech

• English

• Finnish

• French

• German

• Hindi

• Italian

• Japanese

• Korean

• Lithuanian

• Norwegian

• Russian

• Spanish

• Swedish

• Turkish

The offensive content filter scans the crawled properties title, body and ocfcontribution. The last property is not set by the crawlers, but can be used to scan additional content. You can, for example, map custom content to ocfcontribution by using XMLMapper.

Items that are considered pornographic are dropped during processing and appropriate feedback is supplied to the indexing connector.

Metadata Extraction

Certain crawled properties contain metadata from Microsoft Office documents. When an author creates a new document, he or she will typically use a template or another document as a starting point. In many cases the author does not update the metadata, and the metadata will be misleading. For Microsoft Word and Microsoft PowerPoint documents, it is possible to extract date and title information from the content of the document instead. In most cases this generates better metadata.

FAST Search Server 2010 for SharePoint includes an extended metadata extraction stage. When this stage is enabled, the title and date is based on the content of the document, instead of the document metadata.

By default, the extended metadata extraction is enabled. To disable extended metadata extraction, you add the key MetadataExtraction in optionalprocessing.xml, as shown in the following example.

<optionalprocessing>
      <processor name="MetadataExtraction" active="no" />
</optionalprocessing>

If you disable extended metadata extraction, then title and date will be based on the document metadata.

If you use extended metadata extraction, then two crawled properties are created in the item processing pipeline:

• Extracted title:

  • Crawled property name: 302

  • Property set: 012357BD-1113-171D-1F25-292BB0B0B0B0

  • Variant type: 31

  • Mapped to managed property: Title

• Extracted date:

  • Crawled property name: 263

  • Property set: 012357BD-1113-171D-1F25-292BB0B0B0B0

  • Variant type: 64

  • Mapped to managed property: Write

See Also

Concepts

Creating a Custom Property Extractor

Custom XML Item Processing

Configure FAST Search Server for SharePoint to use a Third-Party IFilter

Integrating an External Item Processing Component