Interface IEncodedLineTokens

The result of a line tokenization.

interface IEncodedLineTokens {
endState: IState;
tokens: Uint32Array;
}

endState: IState

The tokenization end state. A pointer will be held to this and the object should not be modified by the tokenizer after the pointer is returned.

tokens: Uint32Array

The tokens on the line in a binary, encoded format. Each token occupies two array indices. For token i:

3322 2222 2222 1111 1111 1100 0000 0000
1098 7654 3210 9876 5432 1098 7654 3210

bbbb bbbb bfff ffff ffFF FFTT LLLL LLLL

L = EncodedLanguageId (8 bits): Use getEncodedLanguageId to get the encoded ID of a language.
T = StandardTokenType (2 bits): Other = 0, Comment = 1, String = 2, RegEx = 3.
F = FontStyle (4 bits): None = 0, Italic = 1, Bold = 2, Underline = 4, Strikethrough = 8.
f = foreground ColorId (9 bits)
b = background ColorId (9 bits)
The color value for each colorId is defined in IStandaloneThemeData.customTokenColors: e.g. colorId = 1 is stored in IStandaloneThemeData.customTokenColors[1]. Color id = 0 means no color, id = 1 is for the default foreground color, id = 2 for the default background.

Properties