StringTokenizer

Java StringTokenizer Example

StringTokenizer class in Java is a class from java.util package. It allows us to break a string into tokens. Here, we show the use of String Tokenizer class, as well as some of its basic operations.

The JDK version we use to compile the source code in this example is OpenJDK 13 and the IDE we use is Eclipse IDE 2020-03.

1. Syntax of the constructors

StringTokenizer provides three constructors listed in the table below. All of them construct a string tokenizer for the specified string. But the string tokenizer behaves in different ways based on the parameters provided to the constructor. Depending on our application, we can choose one of them.

ConstructorDescription
StringTokenizer(String str)The default delimiters will be used and delimiter characters will not be returned as tokens. The default delimiter set includes: " \t\n\r\f" (the space character, the tab character, the newline character, the carriage-return character, and the form-feed character).
StringTokenizer(String str, String delim)Custom delimiters can be specified by providing the delim parameter. Delimiter characters will not be returned as tokens. Note that it is valid to set delim as null when constructing a tokenizaer. But invoking other methods on the tokenizer constructed may result in a NullPointerException.
StringTokenizer(String str, String delim, boolean returnDelims)Custom delimiters can be specified by providing the delim parameter. If the returnDelims flag is true, then the delimiter characters are also returned as tokens. Each delimiter is returned as a string of length one. Otherwise, the delimiter characters are skipped and will not be returned as tokens. Note that it is valid to set delim as null when constructing a tokenizer. But invoking other methods on the tokenizer constructed may result in a NullPointerException.
Table. 1. StringTokenizer Constructors

2. Methods of StringTokenizer

There are six public methods defined in StringTokenizer class as shown in the table below:

MethodDescription
countTokens()Counts the number of tokens the tokenizer has.
hasMoreElements()Same as the hasMoreTokens method.
hasMoreTokens()Returns true if there are more tokens available from this tokenizer’s string; otherwise returns false.
nextElement()Same as the nextToken method, but returns Object rather than String.
nextToken()Returns the next token from this tokenizer.
nextToken(String delim)Returns the next token by using the delimiter specified.
Table. 2. StringTokenizer Public Methods

3. Example of StringTokenizer in Java

Let’s see how to use a StringTokenizer to break a string into tokens in the example below.

import java.util.StringTokenizer;

public class StringTokenizerExample {
    /**
     * @param args
     */
    public static void main(String[] args) {
        String string1 = "It is cool to be a Java code geek!";
        System.out.printf("1. Breaks \"%%s\" by space\n", string1);

        // constructs a string tokenizer with default delimiters
        StringTokenizer tokenizer = new StringTokenizer(string1);
        // loop in order to take all tokens
        while (tokenizer.hasMoreTokens()) {
            // returns a string
            System.out.println(tokenizer.nextToken());
        }

        String string2 = "Java,C++,Basic,Python,Perl";
        System.out.printf("2. Breaks \"%%s\" by comma\n", string2);
        tokenizer = new StringTokenizer(string2, ",");
        System.out.println("The stokenizer includes " + tokenizer.countTokens() + " tokens.");
        while (tokenizer.hasMoreElements()) {
            // returns an object
            Object element = (String) tokenizer.nextElement();
            System.out.println(element.toString());
        }
        
        System.out.printf("3. Breaks \"%%s\" by comma with delimiter returned\n", string2);
        tokenizer = new StringTokenizer(string2, ",", true);
        System.out.println("The stokenizer includes " + tokenizer.countTokens() + " tokens.");
        while (tokenizer.hasMoreElements()) {
            // returns an object
            Object element = (String) tokenizer.nextElement();
            System.out.println(element.toString());
        }
    }
}

Run the example code above and you can see the output as below:

1. Breaks "It is cool to be a Java code geek!" by space
It
is
cool
to
be
a
Java
code
geek!
2. Breaks "Java,C++,Basic,Python,Perl" by comma
The stokenizer includes 5 tokens.
Java
C++
Basic
Python
Perl
3. Breaks "Java,C++,Basic,Python,Perl" by comma with delimiter returned
The stokenizer includes 9 tokens.
Java
,
C++
,
Basic
,
Python
,
Perl

As you can see in the code above, we create three instances of StringTokenizer by using different constructors. In the first one, we use the default space delimiter, while in the second and the third one we choose the comma as the delimiter. In order to return all the tokens, we use a while loop where hasMoreTokens() or hasMoreElements() method is called which indicates whether there are available tokens. In addition, two operations exist (nextToken() and nextElement()) in order to return the value of the next token. The difference between them is that nextElement() method returns an Object instead of a String. Finally, countTokens() method returns the number of the tokens the tokenizer has. In the third tokenizer, we set the returnDelims flag to true to let it return the delimiter as token together with other tokens.

4. StringTokenizer vs String.split()

StringTokenizer was introduced since JDK 1.0. It is retained as a legacy class and mainly for compatibility reasons. So we’d better not to use it in our new code. Instead, the split method of String class introduced since JDK 1.4 provides similar functionality which is recommended. Also, for complex use cases, classes from java.util.regex package can be another option.

In the example below, we use String.split() method to break the same strings into tokens as we’ve done with StringTokenizer before.

public class StringSplitExample {
    /**
     * @param args
     */
    public static void main(String[] args) {
        // split by the space character
        String string1 = "It is cool to be a Java code geek!";
        System.out.printf("1. Splits \"%%s\" by space\n", string1);
        String[] result = string1.split("\\s");
        for (int i = 0; i < result.length; i++) {
            System.out.println(result[i]);
        }

        // split by comma ","
        String string2 = "Java,C++,Basic,Python,Perl";
        System.out.printf("2. Splits \"%%s\" by comma\n", string2);
        result = string2.split(",");
        for (int i = 0; i < result.length; i++) {
            System.out.println(result[i]);
        }
    }
}

Run the example code above and you can see the output as below:

1. Splits "It is cool to be a Java code geek!" by space
It
is
cool
to
be
a
Java
code
geek!
2. Splits "Java,C++,Basic,Python,Perl" by comma
Java
C++
Basic
Python
Perl

5. Download the source code

Download
You can download the full source code of this example here: Java StringTokenizer Example

Last updated on Jun 01st, 2020

Katerina Zamani

Katerina has graduated from the Department of Informatics and Telecommunications in National and Kapodistrian University of Athens (NKUA) and she attends MSc courses in Advanced Information Systems at the same department. Currently, her main academic interests focus on web applications, mobile development, software engineering, databases and telecommunications.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments
Back to top button