The processing of text often consists of parsing a formatted input string. Pnrs Illg is the division of text into a set of discrete parts, or token», which in n certain sequence can COllVer a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lex (lexical analyzer). or scanner. StringTokenizer implements the Enumeration interface. Therefore, given ;11 input string, you can enumerate the individual tokens contained in it using StringTokenizer. To USe Stringtokenizer, you specify an input string and a string The java.util.jar package that contains delimiters. Delimiter» are characters that separate tokens. Each character in the delimiters string is considered a valid delimiter-Cor example sets the delimiters to a comma semicolon, and colon. TIle default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return. The StringTokenizer constructors are shown here:
StringTokenizer 7.cr(String sir, String delimiters) .
StringTokenizer[String sir, String delimiters, boolean dc/imAsToken)
In all versions, str is the string that will be tokenized. In the first version, the default delimiters are used. In the second and third versions, delimiters is a string that specifics the delimiters. In the third version, if delil/lAsTokelt is true, then the deli miters are also returned as tokens when the string is parsed. Otherwise, the deli miters arc not returned. Dellmiters a.re not returned as tokens by the first two forms. Once you have created a StringTokenizcr object, thenexfTokenf ) method is used to extract consecutive tokens. The hasMoreTokens( ) method returns true while there arc more tokens to-be extracted. Since Stringtokenizer implements Enumeration; the has More Elements( ) and nextElement( ) methods are also implemented, and they act the same as hasMoreTokens and next Token( ), respectively.
Here is an example that creates a String Tokenizer to parse “key-value” pairs. Consecutive set~ of “key=value” pairs are separated by a semicolon. The output from this program is shown here:
tit 10 Java: The Complete Reference
author Naughton nnd Schildt