Syntax Highlighting (Managed Package Framework)

To support syntax highlighting, where different elements of a programming language in a source file are displayed in different colors and styles, it is necessary to supply a parser or scanner that can quickly identify the types of lexical elements or tokens in the file. For example, in the programming statement int x = func(y); there are a number of different tokens: x and y are variables, int is a variable type, = is an operator, ( and ) denote a parameter list, and ; is a line delimiter. This statement can be understood more quickly if the various tokens are shown in different colors. This is syntax highlighting, also known as colorization.

Implementation

To support colorization, the managed package framework (MPF) supplies the Colorizer class, which implements the IVsColorizer interface. This class interacts with an IScanner scanner to determine the types of each token on a line and what color they should be. For more information on scanners, see The Language Service Parser and Scanner (Managed Package Framework). The Colorizer class then marks each character of the token with the color information and returns that information to the editor displaying the source file.

The color information returned to the editor is an index into a list of what are called colorable items. Each colorable item specifies a color value and a set of font attributes such as bold or strikethrough. The color value is typically a predefined number that represents a system color. Visual Studio supplies a set of default colorable items that your language service can use. All you need to do is specify the appropriate color index for each token type. However, you can provide a set of custom colorable items and the indices you supply for tokens, then reference your own list of colorable items instead of the default list. You must also set the RequestStockColors registry entry to 0 (or do not specify the RequestStockColors entry at all) to support custom colors. You can set this registry entry with a named parameter to the ProvideLanguageServiceAttribute user-defined attribute. For more information on registering a language service and setting its options, see Registering a Language Service (Managed Package Framework).

Custom Colorable Items

Remember, to support custom colorable items, you must set the RequestStockColors registry entry to 0. To supply your own custom colorable items, you must override the GetItemCount and GetColorableItem method on the LanguageService class. The first method indicates the number of custom colorable items that your language service supports and the second provides access to the list, one item at a time. In your language service's constructor, you create the default list of custom colorable items. vs_current_short automatically handles the case where the user overrides your colorable items with whatever the user selects; all you need to do is supply each colorable item with a name. This name is what appears in the Fonts and Colors property page on the Options dialog box (available from the vs_current_short Tools menu) and this name determines which color a user has overridden. The user's choices are stored in a cache in the registry and accessed by the color name. The Fonts and Colors property page lists all of the color names in alphabetical order so you can group your language service's custom colors by preceding each color name with your language name; for example, "MyLanguage- Comment" and "MyLanguage- Keyword". Or you can group your colorable items by type, "Comment (MyLanguage)" and "Keyword (MyLanguage)". Grouping by language name is preferred.

Note that the first item in your list of colorable items is never referenced- this is for a colorable item index of 0 and vs_current_short always supplies the default text colors and attributes for that item. The easiest way of dealing with this is to supply a placeholder colorable item in your list as the first item.

High Color Colorable Items

Colorable items can also support 24-bit or high color values through the IVsHiColorItem interface. The MPF ColorableItem class supports the IVsHiColorItem interface and the 24-bit colors are specified in the constructor along with the normal colors. See the ColorableItem class for more details. The example below shows how to set the 24-bit colors for keywords and comments. The 24-bit colors are used when 24-bit color is supported on the user's desktop; otherwise, the normal text colors are used.

Remember, these are the default colors for your language; the user can change these colors to whatever they want.

Example

This example shows one way to declare and populate an array of custom colorable items using the ColorableItem class of the managed package framework. This example sets the keyword and comment colors using 24-bit colors which are used where supported.

using Microsoft.VisualStudio.Package;
using Microsoft.VisualStudio.TextManager.Interop;

namespace MyLanguagePackage
{
    public class MyLanguageService : LanguageService
    {
        private ColorableItem[] m_colorableItems;

        MyLanguageService() : base()
        {
            m_colorableItems = new ColorableItem[] {
                new ColorableItem("MyLanguage – Text",
                                  COLORINDEX.CI_SYSPLAINTEXT_FG,
                                  COLORINDEX.CI_SYSPLAINTEXT_BK,
                                  System.Drawing.Color.Empty,
                                  System.Drawing.Color.Empty,
                                  FONTFLAGS.FF_DEFAULT),
                new ColorableItem("MyLanguage – Keyword",
                                  COLORINDEX.CI_MAROON,
                                  COLORINDEX.CI_SYSPLAINTEXT_BK,
                                  System.Drawing.Color.FromArgb(192,32,32),
                                  System.Drawing.Color.Empty,
                                  FONTFLAGS.FF_BOLD),
                new ColorableItem("MyLanguage – Comment",
                                  COLORINDEX.CI_DARKGREEN,
                                  COLORINDEX.CI_LIGHTGRAY,
                                  System.Drawing.Color.FromArgb(32,128,32),
                                  System.Drawing.Color.Empty,
                                  FONTFLAGS.FF_DEFAULT)
                // ...
                // Add as many colorable items as you want to support.
            };
        }
    }
}

The Colorizer class and the Scanner

The base LanguageService class has a method called GetColorizer that creates a new instance of the Colorizer class. When the Colorizer class is instantiated, the scanner that is returned from the GetScanner method is passed to the Colorizer class constructor. The GetScanner method is what you must implement in your own version of the LanguageService class. The Colorizer class uses the scanner to obtain all token color information, so there is no need to override the Colorizer class. You just need to provide a scanner to do the work.

The scanner fills out a TokenInfo structure for every token it finds. This structure contains information such as the span the token occupies, the color index to use, what type is the token, and token triggers (see TokenTriggers). Only the span and color index are needed for colorization by the Colorizer class.

The color index stored in the TokenInfo structure is typically a value from the TokenColor enumeration, which provides a number of named indices corresponding to various language elements such as keywords and operators. If your custom colorable items list matches the items presented in the TokenColor enumeration, then you can just use the enumeration as the color for each token. However, if you have additional colorable items or you do not want to use the existing values in that order, you can arrange your custom colorable items list to suit your needs and return the appropriate index into that list. Just be sure to cast the index to a TokenColor when storing it in the TokenInfo structure; vs_current_short only sees a number to be used as an index.

Example

Here is an abbreviated example of how the scanner might work to identify white space, punctuation, and identifiers (defined as anything that is not white space or punctuation). This example is for illustrative purposes only and does not reflect how a proper scanner should be written. This example does not use the state variable.

using Microsoft.VisualStudio.Package;
using Microsoft.VisualStudio.TextManager.Interop;

namespace MyLanguagePackage
{
    public class MyScanner : IScanner
    {
        //==========================================================
        // Private fields.
        private string m_line;   // line of text to parse
        private int    m_offset; // where next token starts in line

        //==========================================================
        // IVsScanner methods

        public void SetSource(string source, int offset)
        {
            m_line   = source;
            m_offset = offset;
        }


        public bool ScanTokenAndProvideInfoAboutIt(TokenInfo tokenInfo,
                                                   ref int state)
        {
            bool fFoundToken = GetNextToken(m_offset, tokenInfo, ref state);
            if (fFoundToken)
            {
                m_offset = tokenInfo.EndIndex + 1;
            }
            return fFoundToken;
        }

        //==========================================================
        // Private methods.

        private bool GetNextToken(int startIndex, TokenInfo tokenInfo, ref int state)
        {
            bool bFoundToken = false;      // Assume we are done with this line.
            int  index       = startIndex;

            if (index < m_line.Length)
            {
                bFound = true;            // We are not done with this line.
                tokenInfo.StartIndex = index;
                char c = m_line[index];

                if (Char.IsPunctuation(c))
                {
                    tokenInfo.Type     = TokenType.Operator;
                    tokenInfo.Color    = TokenColor.Keyword;
                    tokenInfo.EndIndex = index;
                }
                else if (Char.IsWhitespace(c))
                {
                    do
                    {
                        ++index;
                    }
                    while(index < m_line.Length &&
                          Char.IsWhiteSpace(m_line[index]));
                    tokenInfo.Type     = TokenType.Whitespace;
                    tokenInfo.Color    = TokenColor.Text;
                    tokenInfo.EndIndex = index - 1;
                }
                else
                {
                    do
                    {
                        ++index;
                    }
                    while(index < m_line.Length &&
                          !Char.IsPunctuation(m_line[index]) &&
                          !Char.IsWhiteSpace(m_line[index]));
                    tokenInfo.Type     = TokenType.Identifier;
                    tokenInfo.Color    = TokenColor.Identifier;
                    tokenInfo.EndIndex = index - 1;
                }
            }
            return bFoundToken;
        }
    }
}

See Also

Concepts

Language Service Features (Managed Package Framework)

The Language Service Parser and Scanner (Managed Package Framework)

Registering a Language Service (Managed Package Framework)