Uses of Interface
gov.sandia.cognition.text.token.Tokenizer

Packages that use Tokenizer

Package	Description
gov.sandia.cognition.text.token	Provides text tokenization algorithms.

Uses of Tokenizer in gov.sandia.cognition.text.token

Classes in gov.sandia.cognition.text.token that implement Tokenizer

Modifier and Type	Class and Description
class	AbstractCharacterBasedTokenizer An abstract implementation of a tokenizer that considers each character individually.
class	AbstractTokenizer Abstract implementation of the Tokenizer interface.
class	LetterNumberTokenizer A tokenizer that creates tokens from sequences of letters and numbers, treating everything else as a delimiter.