public abstract class AbstractTokenizer extends AbstractCloneableSerializable implements Tokenizer
Tokenizer
interface. It turns the
tokenize call for a String
into a Reader
.Constructor and Description |
---|
AbstractTokenizer()
Creates a new
AbstractTokenizer . |
Modifier and Type | Method and Description |
---|---|
java.lang.Iterable<Token> |
tokenize(java.lang.String s)
Convert the given string into a corresponding ordered list of tokens.
|
java.lang.Iterable<Token> |
tokenize(Textual textual)
Convert the given string into a corresponding ordered list of tokens.
|
clone
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
clone
public java.lang.Iterable<Token> tokenize(java.lang.String s)
Tokenizer