GPTTokenizer class

Reference

Package:: @microsoft/teams-ai

Tokenizer that uses GPT's cl100k_base encoding. To use GPT-3 encoding, pass in an instance of GPT3Tokenizer.

Methods

decode(number[])
encode(string)

Method Details

decode(number[])

function decode(tokens: number[]): string

Parameters

tokens: number[]

Returns

string

encode(string)

function encode(text: string): number[]

Parameters

text: string

Returns

number[]