Uses of Class
gov.sandia.cognition.text.token.AbstractTokenizer

Packages that use AbstractTokenizer

Package	Description
gov.sandia.cognition.text.token	Provides text tokenization algorithms.

Uses of AbstractTokenizer in gov.sandia.cognition.text.token

Subclasses of AbstractTokenizer in gov.sandia.cognition.text.token

Modifier and Type	Class and Description
class	AbstractCharacterBasedTokenizer An abstract implementation of a tokenizer that considers each character individually.
class	LetterNumberTokenizer A tokenizer that creates tokens from sequences of letters and numbers, treating everything else as a delimiter.