StreamTokenizer

Java StreamTokenizer Example

In this example we will see how to use Java StreamTokenizer class to parse an input stream into tokens. We can use this class to break the InputStream object or an object of type Reader into tokens based on different identifiers, numbers, quoted strings, and various comment styles.

The use of InputStream as the input parameter is deprecated, so we will focus on using a Reader Object as input.

To iterate over the parsed tokens, the method nextToken() can be used.

Lets see this in an example :

JavaStreamTokenizerExample.java

package com.javacodegeeks.example;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

/**
 * 
 * @author anirudh
 *
 */
public class JavaStreamTokenizerExample {

	public static void main(String[] args) throws IOException {
		Reader reader = new StringReader("This is a test string for JCG Stream Tokenizer Example");
		StreamTokenizer tokenizer = new StreamTokenizer(reader);
		while(tokenizer.nextToken()!=StreamTokenizer.TT_EOF){
			System.out.println(tokenizer.sval);
		}
		
	}
}

Output:

test
string
for
JCG
Stream
Tokenizer
Example

As we can see in the above example I used StreamTokenizer.TT_EOF attribute to determine end of string and
tokenizer.sval to get the string value of a token.

There are several other fields as well which can be used to check which type of token is read, what is its value or determine end of line etc.

These fields are :

  • nval field contains the value of the number.
  • TT_EOL is used to determine end of line
  • ttype field contains the type of the token just read.
  • TT_WORD indicates that the token is a word.
  • TT_NUMBER indicates that the token is a number.

Lets see use of these :

JavaStreamTokenizerExample.java

package com.javacodegeeks.example;

import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;

/**
 * 
 * @author anirudh
 *
 */
public class JavaStreamTokenizerExample {

	public static void main(String[] args) throws IOException {
		
		//Read from a file
		FileReader fileReader = new FileReader("/Users/anirudh/test.txt");
		StreamTokenizer filetokenizer = new StreamTokenizer(fileReader);
		while(filetokenizer.nextToken()!=StreamTokenizer.TT_EOF){
			if(filetokenizer.ttype==StreamTokenizer.TT_NUMBER){
				System.out.println(filetokenizer.nval);
			}else if(filetokenizer.ttype==StreamTokenizer.TT_WORD){
				System.out.println(filetokenizer.sval);
			}
		}
	}
}

Output:

Java
code
geeks
is
the
number
1.0
site
of
2014.0
2.0
nd
Line
test

In the above example we read the file using FileReader class which extends InputStreamReader (which in turn extends Reader) to read a file from disc.

The contents of test.txt file are :

Java code geeks is the number 1 site of 2014
2nd Line test

Here, we used TT_NUMBER to identify if the token type was a number and hence used nval to gets its number value. Similarly, we used TT_WORD to check if it is was a String and hence used sval to get its String value.

Download the Eclipse project of this tutorial

This was an example on StreamTokenizer in Java.

Download
You can download the full source code of this example here : JavaStreamTokenizerExample.zip

Anirudh Bhatnagar

Anirudh is a Java programmer with extensive experience in building Java/J2EE applications. He has always been fascinated by the new technologies and emerging trends in software development. He has been involved in propagating these changes and new technologies in his projects. He is an avid blogger and agile enthusiast who believes in writing clean and well tested code.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments
Back to top button