Class WhitespaceTokenizer

Tokenizer that splits on whitespace.

export

Hierarchy

SplittingTokenizer
- WhitespaceTokenizer

Index

Constructors

constructor

Methods

tokenize

Constructors

constructor

new WhitespaceTokenizer(): WhitespaceTokenizer

Overrides SplittingTokenizer.constructor
- Defined in nlp/tokenizer.ts:117
Returns WhitespaceTokenizer

Methods

tokenize

tokenize(query: string): Token[]

Inherited from SplittingTokenizer.tokenize
- Defined in nlp/tokenizer.ts:87
Parameters
- query: string
Returns Token[]

Generated using TypeDoc