StringTokenizer class in Java is a class from java.util package. It allows us to break a string into tokens. Here, we show the use of String Tokenizer class, as well as some of its basic operations.
The JDK version we use to compile the source code in this example is OpenJDK 13 and the IDE we use is Eclipse IDE 2020-03.
1. Syntax of the constructors
StringTokenizer provides three constructors listed in the table below. All of them construct a string tokenizer for the specified string. But the string tokenizer behaves in different ways based on the parameters provided to the constructor. Depending on our application, we can choose one of them.
| Constructor | Description |
|---|---|
| StringTokenizer(String str) | The default delimiters will be used and delimiter characters will not be returned as tokens. The default delimiter set includes: " \t\n\r\f" (the space character, the tab character, the newline character, the carriage-return character, and the form-feed character). |
| StringTokenizer(String str, String delim) | Custom delimiters can be specified by providing the delim parameter. Delimiter characters will not be returned as tokens. Note that it is valid to set delim as null when constructing a tokenizaer. But invoking other methods on the tokenizer constructed may result in a NullPointerException. |
| StringTokenizer(String str, String delim, boolean returnDelims) | Custom delimiters can be specified by providing the delim parameter. If the returnDelims flag is true, then the delimiter characters are also returned as tokens. Each delimiter is returned as a string of length one. Otherwise, the delimiter characters are skipped and will not be returned as tokens. Note that it is valid to set delim as null when constructing a tokenizer. But invoking other methods on the tokenizer constructed may result in a NullPointerException. |
2. Methods of StringTokenizer
There are six public methods defined in StringTokenizer class as shown in the table below:
| Method | Description |
|---|---|
| countTokens() | Counts the number of tokens the tokenizer has. |
| hasMoreElements() | Same as the hasMoreTokens method. |
| hasMoreTokens() | Returns true if there are more tokens available from this tokenizer’s string; otherwise returns false. |
| nextElement() | Same as the nextToken method, but returns Object rather than String. |
| nextToken() | Returns the next token from this tokenizer. |
| nextToken(String delim) | Returns the next token by using the delimiter specified. |
3. Example of StringTokenizer in Java
Let’s see how to use a StringTokenizer to break a string into tokens in the example below.
import java.util.StringTokenizer;
public class StringTokenizerExample {
/**
* @param args
*/
public static void main(String[] args) {
String string1 = "It is cool to be a Java code geek!";
System.out.printf("1. Breaks \"%s\" by space\n", string1);
// constructs a string tokenizer with default delimiters
StringTokenizer tokenizer = new StringTokenizer(string1);
// loop in order to take all tokens
while (tokenizer.hasMoreTokens()) {
// returns a string
System.out.println(tokenizer.nextToken());
}
String string2 = "Java,C++,Basic,Python,Perl";
System.out.printf("2. Breaks \"%s\" by comma\n", string2);
tokenizer = new StringTokenizer(string2, ",");
System.out.println("The stokenizer includes " + tokenizer.countTokens() + " tokens.");
while (tokenizer.hasMoreElements()) {
// returns an object
Object element = (String) tokenizer.nextElement();
System.out.println(element.toString());
}
System.out.printf("3. Breaks \"%s\" by comma with delimiter returned\n", string2);
tokenizer = new StringTokenizer(string2, ",", true);
System.out.println("The stokenizer includes " + tokenizer.countTokens() + " tokens.");
while (tokenizer.hasMoreElements()) {
// returns an object
Object element = (String) tokenizer.nextElement();
System.out.println(element.toString());
}
}
}Run the example code above and you can see the output as below:
1. Breaks "It is cool to be a Java code geek!" by space It is cool to be a Java code geek! 2. Breaks "Java,C++,Basic,Python,Perl" by comma The stokenizer includes 5 tokens. Java C++ Basic Python Perl 3. Breaks "Java,C++,Basic,Python,Perl" by comma with delimiter returned The stokenizer includes 9 tokens. Java , C++ , Basic , Python , Perl
As you can see in the code above, we create three instances of StringTokenizer by using different constructors. In the first one, we use the default space delimiter, while in the second and the third one we choose the comma as the delimiter. In order to return all the tokens, we use a while loop where hasMoreTokens() or hasMoreElements() method is called which indicates whether there are available tokens. In addition, two operations exist (nextToken() and nextElement()) in order to return the value of the next token. The difference between them is that nextElement() method returns an Object instead of a String. Finally, countTokens() method returns the number of the tokens the tokenizer has. In the third tokenizer, we set the returnDelims flag to true to let it return the delimiter as token together with other tokens.
4. StringTokenizer vs String.split()
StringTokenizer was introduced since JDK 1.0. It is retained as a legacy class and mainly for compatibility reasons. So we’d better not to use it in our new code. Instead, the split method of String class introduced since JDK 1.4 provides similar functionality which is recommended. Also, for complex use cases, classes from java.util.regex package can be another option.
In the example below, we use String.split() method to break the same strings into tokens as we’ve done with StringTokenizer before.
public class StringSplitExample {
/**
* @param args
*/
public static void main(String[] args) {
// split by the space character
String string1 = "It is cool to be a Java code geek!";
System.out.printf("1. Splits \"%s\" by space\n", string1);
String[] result = string1.split("\\s");
for (int i = 0; i < result.length; i++) {
System.out.println(result[i]);
}
// split by comma ","
String string2 = "Java,C++,Basic,Python,Perl";
System.out.printf("2. Splits \"%s\" by comma\n", string2);
result = string2.split(",");
for (int i = 0; i < result.length; i++) {
System.out.println(result[i]);
}
}
}Run the example code above and you can see the output as below:
1. Splits "It is cool to be a Java code geek!" by space It is cool to be a Java code geek! 2. Splits "Java,C++,Basic,Python,Perl" by comma Java C++ Basic Python Perl
5. Download the source code
You can download the full source code of this example here: Java StringTokenizer Example
Last updated on Jun 01st, 2020
