In this example we will see how to use Java StreamTokenizer class to parse an input stream into tokens. We can use this class to break the InputStream object or an object of type Reader into tokens based on different identifiers, numbers, quoted strings, and various comment styles.
The use of InputStream as the input parameter is deprecated, so we will focus on using a Reader Object as input.
To iterate over the parsed tokens, the method nextToken() can be used.
Lets see this in an example :
JavaStreamTokenizerExample.java
package com.javacodegeeks.example;
import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;
/**
*
* @author anirudh
*
*/
public class JavaStreamTokenizerExample {
public static void main(String[] args) throws IOException {
Reader reader = new StringReader("This is a test string for JCG Stream Tokenizer Example");
StreamTokenizer tokenizer = new StreamTokenizer(reader);
while(tokenizer.nextToken()!=StreamTokenizer.TT_EOF){
System.out.println(tokenizer.sval);
}
}
}
Output:
test string for JCG Stream Tokenizer Example
As we can see in the above example I used StreamTokenizer.TT_EOF attribute to determine end of string and
tokenizer.sval to get the string value of a token.
There are several other fields as well which can be used to check which type of token is read, what is its value or determine end of line etc.
These fields are :
- nval field contains the value of the number.
- TT_EOL is used to determine end of line
- ttype field contains the type of the token just read.
- TT_WORD indicates that the token is a word.
- TT_NUMBER indicates that the token is a number.
Lets see use of these :
JavaStreamTokenizerExample.java
package com.javacodegeeks.example;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;
/**
*
* @author anirudh
*
*/
public class JavaStreamTokenizerExample {
public static void main(String[] args) throws IOException {
//Read from a file
FileReader fileReader = new FileReader("/Users/anirudh/test.txt");
StreamTokenizer filetokenizer = new StreamTokenizer(fileReader);
while(filetokenizer.nextToken()!=StreamTokenizer.TT_EOF){
if(filetokenizer.ttype==StreamTokenizer.TT_NUMBER){
System.out.println(filetokenizer.nval);
}else if(filetokenizer.ttype==StreamTokenizer.TT_WORD){
System.out.println(filetokenizer.sval);
}
}
}
}
Output:
Java code geeks is the number 1.0 site of 2014.0 2.0 nd Line test
In the above example we read the file using FileReader class which extends InputStreamReader (which in turn extends Reader) to read a file from disc.
The contents of test.txt file are :
Java code geeks is the number 1 site of 2014 2nd Line test
Here, we used TT_NUMBER to identify if the token type was a number and hence used nval to gets its number value. Similarly, we used TT_WORD to check if it is was a String and hence used sval to get its String value.
Download the Eclipse project of this tutorial
This was an example on StreamTokenizer in Java.
You can download the full source code of this example here : JavaStreamTokenizerExample.zip
