A "manual" provider of tokens, returning tokens in a binary form.

interface EncodedTokensProvider {
    getInitialState(): IState;
    tokenize?(line, state): ILineTokens;
    tokenizeEncoded(line, state): IEncodedLineTokens;


  • The initial state of a language. Will be the state passed in to tokenize the first line.

    Returns IState

  • Tokenize a line given the state at the beginning of the line.


    Returns ILineTokens