Class SplittingTokenizer

Tokenizing base-class that will split on the given RegExp.

export
abstract
implements: {Tokenizer}

Hierarchy

SplittingTokenizer
- WhitespaceTokenizer

Implements

Tokenizer

Index

Constructors

constructor

Methods

tokenize

Constructors

constructor

new SplittingTokenizer(pattern: RegExp): SplittingTokenizer

- Defined in nlp/tokenizer.ts:78
Creates an instance of SplittingTokenizer.

memberof

SplittingTokenizer

Parameters
- pattern: RegExp
  
  The pattern to split on.
Returns SplittingTokenizer

Methods

tokenize

tokenize(query: string): Token[]

Implementation of Tokenizer.tokenize
- Defined in nlp/tokenizer.ts:87
Tokenizes a string.

Parameters
- query: string
Returns Token[]

The tokens.

Generated using TypeDoc