A "manual" provider of tokens.
The initial state of a language. Will be the state passed in to tokenize the first line.
Tokenize a line given the state at the beginning of the line.
A "manual" provider of tokens.