Interface for a meaningful chunk of text, called a token.
Interface for a class that converts strings into tokens.
An abstract implementation of a tokenizer that considers each character individually.
Abstract implementation of the
A default implementation of the
A tokenizer that creates tokens from sequences of letters and numbers, treating everything else as a delimiter.