Options
All
  • Public
  • Public/Protected
  • All
Menu

Interface Tokenizer

Tokenizer interface for strings.

export
interface

Tokenizer

Hierarchy

  • Tokenizer

Implemented by

Index

Methods

Methods

tokenize

  • tokenize(query: string): Token[]

  • Tokenizes a string.

    memberof

    Tokenizer

    Parameters

    • query: string

      The string to tokenize.

    Returns Token[]

    The tokens.

Generated using TypeDoc