gov.sandia.cognition.text.token

Class AbstractTokenizer

java.lang.Object

gov.sandia.cognition.util.AbstractCloneableSerializable

gov.sandia.cognition.text.token.AbstractTokenizer

All Implemented Interfaces:

Tokenizer, CloneableSerializable, java.io.Serializable, java.lang.Cloneable

Direct Known Subclasses:

AbstractCharacterBasedTokenizer

public abstract class AbstractTokenizer
extends AbstractCloneableSerializable
implements Tokenizer

Abstract implementation of the Tokenizer interface. It turns the tokenize call for a String into a Reader.

Since:

3.0

Author:

Justin Basilico

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
AbstractTokenizer() Creates a new AbstractTokenizer.

Method Summary

Modifier and Type	Method and Description
java.lang.Iterable<Token>	tokenize(java.lang.String s) Convert the given string into a corresponding ordered list of tokens.
java.lang.Iterable<Token>	tokenize(Textual textual) Convert the given string into a corresponding ordered list of tokens.

Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable

clone

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface gov.sandia.cognition.text.token.Tokenizer

tokenize

Methods inherited from interface gov.sandia.cognition.util.CloneableSerializable

clone

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()

Creates a new AbstractTokenizer.

Method Detail

tokenize

public java.lang.Iterable<Token> tokenize(java.lang.String s)

Description copied from interface: Tokenizer

Convert the given string into a corresponding ordered list of tokens.

Specified by:

tokenize in interface Tokenizer

Parameters:

s - The string to tokenize.

Returns:

The ordered list of tokens.

tokenize

public java.lang.Iterable<Token> tokenize(Textual textual)

Description copied from interface: Tokenizer

Convert the given string into a corresponding ordered list of tokens.

Specified by:

tokenize in interface Tokenizer

Parameters:

textual - The object to tokenize.

Returns:

The ordered list of tokens.