From site column to managed property - What's up with that?

  • Article

This is a blog post in the series “How to set up a product-centric website in SharePoint Server 2013”.  In this series, I'll use data from a fictitious company called "Contoso" to show you how to use search features to set up a website based on product catalog data.  

For an overview of the blog posts in this series, go to How to set up a product-centric website in SharePoint Server 2013.

 

Quick overview

In Stage 4: Set up search and enable the crawling of your catalog content, I showed you how to set up search and enable crawling of your catalog content. In this blog post, I'll explain what happens to site columns during crawl.

When a catalog is crawled, the catalog content is added to the search index, where the site columns are represented as crawled and managed properties. Now, I just threw a lot of new words at you, so, let me explain more about the search index and crawled and managed properties.

 

In this blog post, we'll learn:

 

 

About the search index

 Let's start by going back to our overview diagram.

Overview of Contoso architecture

1. When a content source is crawled, its contents and metadata are added to the search index. In our scenario, that means that the content from the Products catalog is added to the search index. 

2. To display content on the publishing portal (the Contoso website), Search Web Parts are used. Search Web Parts contain a query, for example, "show all MP3 players that have the color red" (note: the query is not formulated in such common language, but for now, let's just keep it simple).  So when users navigate to a page that contains a Search Web Part, the query for red MP3 players is automatically sent to the search index. The query result is returned from the search index, and all red MP3 players are shown in the Search Web Part on the page.

 

What is important to understand, is that the content that we add to the search index, determines which search results can be returned when queries are issued from Search Web Parts. Simply put; if the content from our Products catalog isn't added to the search index, we can't display any catalog content on the Contoso website.

 

 

About crawled and managed properties

During a crawl, the contents and metadata of the items are represented as crawled properties. In our scenario, items are the site columns and the values that are stored in the Products catalog. The reason you need to know about crawled properties, is because we'll later use crawled properties when configuring refiners.

Even though crawled properties contain values, they are "useless" without a partner. This partner is a managed property. By creating a mapping between a crawled property and a managed property, we'll be able to display content on Our Contoso website.

The reason you need to know about managed properties, is because we'll later use managed properties when configuring queries in Search Web Parts, and when defining how search results should be displayed on a web page.

I'll explain both the use of crawled and managed properties in more detail along with examples in later blog articles. For now, it's important that you understand that we'll use crawled and managed properties, and NOT the site columns from the Products catalog when querying for and displaying content on Our Contoso website.

About automatically created crawled and managed properties 

SharePoint comes with a number of default managed properties that are mapped to default crawled properties, for example, “Title”.

In previous versions of SharePoint, whenever a new item was discovered during a crawl, users would have to manually create a new managed property, and map this to the corresponding crawled property.

In SharePoint Server 2013, you can use automatically created crawled and managed properties. They make the lives of catalog owners a lot easier.

When new content is added to a catalog (what we did in Stage 2: Import list content into the Product Catalog Site Collection), a lot of new items will be discovered during a crawl (what we did in Stage 4: Set up search and enable the crawling of your catalog content). To avoid having to create a new managed property, and map this to the corresponding crawled property, SharePoint Server 2013 does this for you. When a catalog's crawled, SharePoint Server 2013 automatically creates a crawled property for each site column in the catalog. SharePoint Server 2013 also automatically creates a managed property for the site column, and adds a mapping between the two.

Important note:
Only site columns that contain values automatically become managed properties when they're crawled.  Regular columns do not!

So, that is a sweet deal!

However, remember how I mentioned that you will use crawled- and managed properties in later stages when configuring refiners and queries in Search Web Parts? Well, in order to do that, it's important that you understand how the names of these automatically created crawled and managed properties are constructed.

 

 

About the naming convention for automatically created crawled and managed properties

The name of an automatically created property is a combination of the site column name, and the site column type.  This naming convention is a tricky area to understand, so I'll start with a simple example, with two site columns, and then I'll explain the rules that apply to all site column types.

Naming convention for two site column types

Naming convention for two site columns

1. In the Products list, we have:

  • A site column named "Contoso Color" of type "Single line of text"
  • A site column named "Contoso Weight" of type "Number"

2. During a crawl, a crawled property is automatically created for the site column. The name of this property is generated by removing spaces from the site column name, and adding a prefix. The prefix that is added varies depending on the site column type.
So, in this example, we get the following crawled properties:

  • ows_q_TEXT_ContosoColor
  • ows_q_NMBR_ContosoWeight

3. A managed property is then automatically created, which is mapped to the corresponding crawled property. The name of this property is generated by removing spaces from the site column name, and adding a suffix. The suffix that is added varies depending on the site column type. 
So, in this example, we get the following managed properties:

  • ContosoColorOWSTEXT
  • ContosoWeightOWSNMBR

That was simple, right? However, the naming convention does vary depending on the site column type. 

Naming convention for all site column types

For the creation of an automatically created crawled property name, the following apply:

  • Spaces are removed from the site column name.
  • For site columns of type Publishing HTML and Multiple line of text, the following prefix is added to the site column  name: ows_r_<four letter code>_
  • For site columns of type Managed Metadata, the following prefix is added to the site column  name: ows_taxId_
  • For all other site column types, the following prefix is added to the site column name: ows_q_<four letter code>_

 

For the creation of an automatically managed property name, the following apply:

  • Spaces are removed from the site column name
  • For site columns of type Managed Metadata, the following prefix is added to the site column name: ows_taxId
  • For all other site column types, the following suffix is added to the site column name: OWS <four letter code>

 

If you need more info, there's an overview table with examples for all site column types in the TechNet article Automatically created managed properties in SharePoint 2013.

So, that might not have been as straightforward as our simple example, and there is one exception to the rule…

Naming convention for the default site columns in the Products list

This exception applies to the six site columns that by default are associated with the Products list in the Product Catalog Site Collection, as shown in the table right below:

Site column name Crawled property name Managed property name
Title Title Title
Item Number ows_q_TEXT_ProductCatalogItemNumber ProductCatalogItemNumberOWSTEXT
Group Number ows_q_TEXT_ProductCatalogGroupNumber  ProductCatalogGroupNumberOWSTEXT
Language Tag ows_q_CHCS_ProductCatalogLanguageTag  ProductCatalogLanguageTagOWSCHCS
Item Category ows_taxId_ProductCatalogItemCategory  owstaxIdProductCatalogItemCategory
Rollup Image ows_r_IMGE_PublishingRollupImage PublishingImage

Now, after all that theory, let's move on and get something concrete done! The real fun will start when I show you how to connect a publishing site to a catalog.  Stay tuned :-)

 

 

Next blog article in this series
Stage 5: Connect your publishing site to a catalog

 

Additional resources 

Comments

  • Anonymous
    January 01, 2003
    Hi Anna,Why do site lookup columns not have a corresponding managed property value (like other site columns with values in them) ? Also, how do I add a managed property value to these columns? I have spent a day researching this and the best explanation I found was confusing.Lookup columns in my sharepoint site are important values and need to be included in the search of the list they reside in ans not the reference table that they come from.Any help would be appreciatedThanks,Luis

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    Hi Steve, Automatically generated managed properties are always of type text, so this is not a bug. In the TechNet article “Automatically created managed properties in SharePoint Server 2013” (technet.microsoft.com/.../jj613136.aspx), there is a table called “Examples of automatically created crawled property names and managed property names for site columns without special characters”. In the “Data format” column of this table, you can see how the data formats for automatically created properties can be used. Regarding the two different crawled properties that you mention, yes this can be confusing. Crawled properties that begin with either ows_r<four letter code>, ows_q<four letter code>, or ows_taxId are automatically created crawled properties. Crawled properties that begin with ows_ are “normal” crawled properties. In your case where you want to use a property in a greater than or less than comparison, you should create a new managed property, and map the “normal” crawled property (the one that starts with ows_) to your newly created managed property. Also, if you want to use automatically created managed properties as refiners, there are some data format considerations to be aware of. For more information about this, see the TechNet article “Configure refiners and faceted navigation in SharePoint Server 2013” (technet.microsoft.com/.../jj679902.aspx) Hope this helps. Bella

  • Anonymous
    January 01, 2003
    Hi Daoud, The behavior you are seeing is correct. That is, it is correct that you can only see one ows_column123 crawled property. A query on the managed property column123OWSTEXT (the automatically generated managed property) will return search results from both sites. If you want to return search results from only one site, you have to use one of the “empty” managed properties (see the table "Default unused managed properties” in this article: http://technet.microsoft.com/en-us/library/jj219667.aspx). On your site, go to Site settings --> Search Schema, and map the crawled property ows_column123 to for example RefinableString01. When you use the RefinableString 01 managed property in a query, it’ll narrow down the search results to the site. Hope this helps,
    Bella

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    Hi Jyothsna, Sounds like you are on the right track :-) Bella

  • Anonymous
    January 01, 2003
    Hi Sakti,

    There is a different between List columns and Site columns. List columns won't become automatically generated managed properties. But Site columns will.

    So don't go to List settings and create the list column, but go to Site Settings --> Site Columns, and then create a Site column.

    Hope this helps,
    Bella

  • Anonymous
    January 01, 2003
    Hi Jyothsna, It is recommended that you plan your taxonomy before you connect your publishing site to a catalog. However, if you during your catalog connection selected “Pin terms to site navigation” (blogs.technet.com/.../stage-5-connect-your-publishing-site-to-a-catalog.aspx), any changes you make to your taxonomy term set will be reflected on your publishing site. Hope this helps. Bella

  • Anonymous
    January 01, 2003
    CSWP stands for Content Search Web Part. I talk about this Web Part and also show an example of how you can configure the query in a later post:  blogs.technet.com/.../stage-9-configure-the-query-in-a-content-search-web-part-on-a-category-page.aspx And yes, you can use the query ContosoColorOWSTEXT:red. It will return all items that contain the value "red" for the site column Contoso Color. Hope this helps. Bella

  • Anonymous
    January 01, 2003
    Hi Luis,

    From your description, it sounds like you did everything correct.

    However, was there a reason why you chose to create a new managed property? That is, did you try to use the default ModifiedBy managed property?

    Bella

  • Anonymous
    January 01, 2003
    Hi Parham, Thanks for the feedback :-) I have a few follow up questions:

  • The irrelevant managed properties you mention, did you delete them in Central Administration?
  • Did you also delete the site columns you no longer wanted?
  • For the new site columns, did you give them the exact same Title as the site columns you deleted?
  • When you set the flag to reindex the catalog, is continuous crawling enabled?
  • When you say "I haven't gotten the managed properties that I expect", what kind of properties are you getting? Bella
  • Anonymous
    January 01, 2003
    Hello Bella,I'm interested in how the search handles site columns with the exact same name, in other words, is there a mapping of the site column to a crawled property? I have 2 different site collections. I created a site column in each with the exact same name (column123). I performed a full crawl. I only see one ows_column123. Is this correct? Shouldn't there be two ows_xxxxxx columns? If I perform a search query will the result include content from both site columns?Thanks.Daoud
  • Anonymous
    January 01, 2003
    Hi Andreas, You are correct that a managed property name is based on the internal name. This blog article is intended for users that work in the UI, and they are not able to see the internal name.   That being said, when working in the UI and creating a site column, the internal name is created based on the site column name (yes, site column name is the same as DisplayName). From the site column name, spaces and special characters such as ø, æ and ü are removed to create the internal name. So, in a case where a user creates a site column and mistypes the site column name, for example “Product namee”, the internal name for that property will be “Productnamee”. If the user corrects the site column name to “Product name”, the internal name will remain “Productnamee”. Regarding your questions:
  1. No, you will not have one managed property for each localized version. You have to make sure that the managed properties that you create are based on the same internal name, and then the DisplayName can wary across language versions.
  2. No. When you change the DisplayName, it will not effect the internal name. When creating internal names, you should try to use alphanumeric characters only. That will make it easier to find the properties later on. Hope this helps. Bella

  • Anonymous
    January 01, 2003
    Hi Luis, Not sure if you meant this for someone else (Anna), or if I've missed something. Either way, I’ll try to answer your question. For site columns of type Lookup, automatically created managed properties are not created. However, a crawled property is created, and you can use this to create your own managed property. For example, in a list you have a Lookup site column called CitiesNorway. When you crawl this list, a crawl property calledows_CitiesNorway will be created. In Central Administration you can then create a new managed property, for exampleCitiesInNorway, that you map to the crawled property ows_CitiesNorway. After recrawling, you can use the new managed property,CitiesInNorway. to display content. For more information on how to create a new managed property, see this article: http://technet.microsoft.com/en-us/library/jj219667.aspx#proc2) If you don’t have access to Central Administration, you can use one of the refiner-enabled managed properties and map this to the crawled propertyows_CitiesNorway. I describe how you can to this in another blog: http://blogs.technet.com/b/tothesharepoint/archive/2013/06/19/stage-14-configure-refiners-for-faceted-navigation.aspx#HowToMap. Hope this helps,
    Bella

  • Anonymous
    January 01, 2003
    Hi Jorgete, Could you tell me a bit more about what content source and file types you are trying to crawl? Thanks, Bella

  • Anonymous
    January 01, 2003
    Hi Gerd, I am glad you like the series :-) No, catalogs are not limited to products. They can also be used for articles or any other type of information that you want to publish according to a specific structure or hierarchy.  If you want to see an example, you can take a look at at this case study: technet.microsoft.com/.../jj822912.aspx. It shows how the company Mavention used catalogs and cross-site publishing for their website. The Mavention website consists of different type of articles. Regarding you question about using separate catalogs. Without knowing anything about your content, I would say that it does not make scene to use separate catalogs. You can store all your article information in one catalog, and use the tags from your term set to tag each individual article. Hope this helps. Bella

  • Anonymous
    January 01, 2003
    Hi Jyothsna, Did you change the display template so that it'll show the newly created managed property? For information about about display templates, see msdn.microsoft.com/.../jj945138.aspx Bella

  • Anonymous
    January 01, 2003
    Hello Mr Stone, No, you can search for any managed property in your catalog. If you want to manually build a query, when you configure the query in a CSWP, on the BASIC tab, select Switch to Advanced Mode. In advanced mode you can use the Keyword filter and Property filter menus to build your query. Hope this helps. Bella

  • Anonymous
    January 01, 2003
    Hi Robert,

    List columns that are created in the list will not become automatically generated managed properties. So you have to create your site columns through Site Settings --> Site Columns, and then add them to your list.

    Bella

  • Anonymous
    January 01, 2003
    Hi Mickey,

    Just to see what happens, have you tested to see what happens if you mark the property as Queryable and make a query directly against the property?

    Bella

  • Anonymous
    January 01, 2003
    Hi Daryl, Strange that you can’t see the managed property for your Managed Metadata site column..??….. Can you see managed properties for other site column types? And have you tested with another Managed Metadata site column? Bella

  • Anonymous
    January 01, 2003
    Hi Sri,

    Please contact MS Support with your scenario.

    Thanks,
    Bella

  • Anonymous
    January 01, 2003
    Hi kourosh, I am not an expert on the Search Rest API, but I can certainly try to find the answers to your questions. However, I have a few questions about your scenario, so good if you can help me understand this a bit better.

  • From what I understand, you want to use managed navigation with faceted navigation (different refiners for different terms), correct?
  • Do you have 30 term sets, or one term set with 30 terms?
  • The managed properties and crawled properties you have created, are these the properties you want to add as refiners for faceted navigation?
  • Can you explain a bit more what you mean by “retrieve terms based on their refiners? Thanks, Bella

  • Anonymous
    January 01, 2003
    Hi Kelly,

    Yes, "contains values" means that the site column has to be filled in at least once.

    For example, say you have a site column named Color that you've added to a list. In order for the Color site column to automatically become a managed property, you have to have added at least one item to your list, and that item has to have a value for Color, for example blue. If you just add the list item without entering a value for Color, then no automatically created property will be created for Color.

    Hope this helps,
    Bella

  • Anonymous
    January 01, 2003
    Hi Elina,

    It's a bit difficult to say what could be causing your issue, but have you made sure that your content has been correctly crawled after it's been updated?

    Bella

  • Anonymous
    January 01, 2003
    Hi Elina,

    It's a bit difficult to say what could be causing your issue, but have you made sure that your content has been correctly crawled after it's been updated?

    Bella

  • Anonymous
    March 10, 2013
    Hi Bella, Thank you for your fantastic article about XSP, it's very well-written document on XSP. I have a question about managed properties. I was wondering if you could spare sometime and help me to solve my problem. Due to changes in the site columns, I created my authoring site for couple of times and crawled the sites few times and as consequence I've few irrelevant managed & crawled properties. I tried to remove the extra managed properties and assumed that by reindexing my list, I would automatically get the right managed properties back, however, after running the crawler on my site for few times I haven’t got the managed properties that I expect (which exist on my list!). Could you please help me how I can get the managed properties back? Cheers, Parham

  • Anonymous
    April 25, 2013
    Hi and thanks for this helpful post. I have a problem maybe you can help me out. I don't have any Site Column, any Catalog Site Collection or list. I have about 30 Terms and TermSets (in MMS). Now I want to use them as a search based menu. by using Powershell I have created Managed Properties, Crawled Properties and mapped them to each other. I have enabled the Terms and TermSets for Faceted Navigation and added the Managed Properties to the Terms as Refiners. run Full Crawl. Now I want to retrieve Terms based on their Refiners by using Search Rest API. is it possible? because when I run following queries get no rows. http://thesite/_api/search/query?querytext='ManagedPropertyName:TermName' http://theSite/_api/search/query?querytext='termName'&refiners='ManagedPropertyName' and many other options but no results. it seems that you have to have Site Columns and Catalogs. thanks again.

  • Anonymous
    May 02, 2013
    It appears to be a bug when the managed property that is auto generated is always a text type even if the site column is date or number. So if I try to use the managed property with an operator of <,> or range it produces a syntax error. Also, there are two crawled properties one with ows_q and one just with ows_. This has been a source of confusion for many search administrators. Finally, in SP2010 auto generation of managed properties was an option on the crawled property category, this is no longer available and it seems as if administrators cannot turn this behavior off for site columns. It would be nice to have this feature back.

  • Anonymous
    June 15, 2013
    Hi Bella, This post was so helpful to start with cross-site publishing feature. I have followed the steps provided and created a sample site. I'm facing one issue with Content search web part. I have created color and weight site columns using the powershell and data from these columns shows perfect. Afterwards I have created one text column as a sitecolumn from the UI and added to the Product with Image content type. Managed Property got created for this column. The problem comes here, its not showing the data present in the newly created column in my content search webparts. I had re-index the list and done full-crawl. Can you please help me out for any setting that needs to be done for getting the information. Thanks.

  • Anonymous
    June 16, 2013
    Hi Bella, Thanks for considering my issue. I have created with new display template and with existing template(Image with 3 lines). In the OOB template I have tried by changing the property mapping. Please let me know if I need to verify any thing else. Thanks

  • Anonymous
    June 20, 2013
    Thanks for you response. What ever previous mentioned issue related to site column has been resolved. This issue came us because of less space in search application related DB. After moving to the new server it has been solved. I need your suggestion related to Taxonomy. Do we need to decide our Taxonomy before connecting to Portal site. Any modification to the term set, terms or addition of new term sets, terms will reflect in the portal site. Jyothsna

  • Anonymous
    June 27, 2013
    The comment has been removed

  • Anonymous
    September 03, 2013
    The comment has been removed

  • Anonymous
    September 30, 2013
    Hello, Thanks for your post! I have a question for this. Can I only Search on [Contoso Color] field? Example: I only return all document that [Contoso Color]='red'. Which query text can I use? Thank you!

  • Anonymous
    September 30, 2013
    Thanks Bella. What does CSWP stand for? Is it a tool to manual build query (as CAML query builder tool)? And, can I use query like ContosoColorOWSTEXT:red ?

  • Anonymous
    October 04, 2013
    This is a great series of articles and I'm learning alot. I have a question regarding catalogs. Are they just used for products per say or can the be used for any type of information? I've been tasked to build a magazine site with sharepoint 2013. I will need to rollup a variety of specific article pages that have taxonamy terms associated to them when the page is created. Would it make sense to create catalogs to store the information from the article pages in seperate catalogs by term? I guess I'm a bit confused what gets stored in catalogs and if article pages can or should be stored there for the purpose of rollup. Thanks Gerd

  • Anonymous
    January 21, 2014
    This is a blog post in the series “ How to display Recommendations and Popular items on a SharePoint

  • Anonymous
    February 16, 2014
    Great post, quite informative thanks. However, I have a question regarding Managed Metadata site columns. I created a site column of Managed Metadata type. Added it to Enterprise Wiki library, and created a few pages & set the values in the columns. I have run a full crawl & also an incremental but still cannot see my column. Any idea what could be the issue?

  • Anonymous
    February 25, 2014
    Yesterday, Microsoft announced that search driven publishing features are available in SharePoint Online

  • Anonymous
    March 15, 2014
    Hello Bella,
    Great post. I do have a comment. Back to internal names. When we create a site column such as "Policy Number" the internal name is "Policy_x0020_Number". I've actually got around this by creating all columns through PowerShell and just removing the spaces and leaving the space in the display name. Regardless, we have many crawled properties which contain x0200 even after resetting the index.

    Now a question. How do I use ModifiedBy as a refiner? Keep in mind that I have created a managed property, made it definable, mapped ows_modifiedby (and any crawled property that was close to that) I mapped it to "Office:8" and added it the refiners in the web part. It never shows up with any content. Reindexed several times with no luck.

    Luis

  • Anonymous
    April 09, 2014
    This is a blog post in the series, “ Set up a Search Center in SharePoint 2013 ”.
    In the

  • Anonymous
    April 16, 2014
    Great Series, Lots of knowledge in it and thanks for sharing this. I am also in the list of followers, every thing seems to be working but only thing is that images are not showing up in the publishing site. I have not used power shell script you mentioned but manually set up a quick term set similar to the image shown in your blog. on the publishing site when I click on "Computers" it shows me all items under term computer and when i click on laptop it shows me all laptops but no images , upon selecting one laptop i can see all details like tag , item but No Images ,what am I missing here ? any idea ?
    i have uploaded images in Product list and save then in a separate image library, I also confirmed that they are part of the index now as i can search them and it shows up with image in search result.aspx page.

    Thanks

  • Anonymous
    May 01, 2014
    Great article, answered quite a few questions. I have created a new site column, added to a content type and populated the new column with data. I have marked is searchable, retrievable and the crawled prop is indexed. I do not get this column back in the search properties nor can I find things based on a search. Any thought on what I may have missed?

  • Anonymous
    May 07, 2014
    When you say site columns can only be used, can these be columns created in a list on the fly or do they have to be created in Site Settings>Site Columns and added to a list?

  • Anonymous
    October 13, 2014
    In this series of "how to's" blog posts I'll describe how you can use SharePoint Server

  • Anonymous
    January 07, 2015
    How to find Crawl Properties for Calculated Columns. I am unable to find one.

  • Anonymous
    March 23, 2015
    The comment has been removed

  • Anonymous
    April 03, 2015
    Thank you, Bella!