gov.sandia.cognition.text.token

Class LetterNumberTokenizer

java.lang.Object

gov.sandia.cognition.util.AbstractCloneableSerializable

gov.sandia.cognition.text.token.AbstractTokenizer

gov.sandia.cognition.text.token.AbstractCharacterBasedTokenizer

gov.sandia.cognition.text.token.LetterNumberTokenizer

All Implemented Interfaces:

Tokenizer, CloneableSerializable, java.io.Serializable, java.lang.Cloneable

public class LetterNumberTokenizer
extends AbstractCharacterBasedTokenizer

A tokenizer that creates tokens from sequences of letters and numbers, treating everything else as a delimiter.

Since:

3.0

Author:

Justin Basilico

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
LetterNumberTokenizer() Creates a new LetterNumberTokenizer.

Method Summary

Modifier and Type	Method and Description
boolean	isTokenMember(char c) Determines if the given character is considered to be part of a token.

Methods inherited from class gov.sandia.cognition.text.token.AbstractCharacterBasedTokenizer

tokenize

Methods inherited from class gov.sandia.cognition.text.token.AbstractTokenizer

tokenize, tokenize

Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable

clone

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface gov.sandia.cognition.util.CloneableSerializable

clone

Constructor Detail

LetterNumberTokenizer

public LetterNumberTokenizer()

Creates a new LetterNumberTokenizer.

Method Detail

isTokenMember

public boolean isTokenMember(char c)

Description copied from class: AbstractCharacterBasedTokenizer

Determines if the given character is considered to be part of a token.

Specified by:

isTokenMember in class AbstractCharacterBasedTokenizer

Parameters:

c - A character.

Returns:

True if the character can be part of a token; otherwise, false.