LexicalTokenizer type

Contains the possible cases for Tokenizer.

type LexicalTokenizer =
  | ClassicTokenizer
  | EdgeNGramTokenizer
  | KeywordTokenizer
  | MicrosoftLanguageTokenizer
  | MicrosoftLanguageStemmingTokenizer
  | NGramTokenizer
  | PathHierarchyTokenizer
  | PatternTokenizer
  | LuceneStandardTokenizer
  | UaxUrlEmailTokenizer