gov.sandia.cognition.text.token

Interface Tokenizer

All Superinterfaces:

java.lang.Cloneable, CloneableSerializable, java.io.Serializable

All Known Implementing Classes:

AbstractCharacterBasedTokenizer, AbstractTokenizer, LetterNumberTokenizer

public interface Tokenizer
extends CloneableSerializable

Interface for a class that converts strings into tokens.

Since:

3.0

Author:

Justin Basilico

Method Summary

Modifier and Type	Method and Description
java.lang.Iterable<Token>	tokenize(java.io.Reader reader) Converts the string from the given reader into an ordered list of tokens.
java.lang.Iterable<Token>	tokenize(java.lang.String s) Convert the given string into a corresponding ordered list of tokens.
java.lang.Iterable<Token>	tokenize(Textual textual) Convert the given string into a corresponding ordered list of tokens.

Methods inherited from interface gov.sandia.cognition.util.CloneableSerializable

clone

Method Detail

tokenize

java.lang.Iterable<Token> tokenize(java.lang.String s)

Convert the given string into a corresponding ordered list of tokens.

Parameters:

s - The string to tokenize.

Returns:

The ordered list of tokens.

tokenize

java.lang.Iterable<Token> tokenize(java.io.Reader reader)

Converts the string from the given reader into an ordered list of tokens.

Parameters:

reader - The reader to tokenize the data from.

Returns:

The ordered list of tokens.

tokenize

java.lang.Iterable<Token> tokenize(Textual textual)

Convert the given string into a corresponding ordered list of tokens.

Parameters:

textual - The object to tokenize.

Returns:

The ordered list of tokens.