StreamTokenizer stops when it hits a slash

Question

Consider this program , found on tutorial points (where br is a BufferedReader object):

boolean eof = false;
// Parse incoming request
StreamTokenizer st = new StreamTokenizer(br);
do {
    int token = st.nextToken();

    switch (token) {
        case StreamTokenizer.TT_EOF:
            System.out.println("End of File encountered.");
            eof = true;
            break;

        case StreamTokenizer.TT_EOL:
            System.out.println("End of Line encountered.");
            break;

        case StreamTokenizer.TT_WORD:
            System.out.println("Word: " + st.sval);
            break;

        case StreamTokenizer.TT_NUMBER:
            System.out.println("Number: " + st.nval);
            break;

        default:
            System.out.println((char) token + " encountered.");

            if (token == '!') {
                eof = true;
            }
    }
    } while (!eof);

This snippet works on strings like "Hello this is a string" where it will print out:

Word: Hello

Word: this

Word: is

Word: a

Word: string

But if I do something like this: "Hello /this is a string" It will only print out

Word: Hello

Why is this, and how can i solve it?

Answer 1

The forward slash / is not treated as an ordinary character in the StreamTokenizer class. You can change that default behavior by using the ordinaryChar method:

import java.io.*;

public class StreamTokenizerSO {
    public static void main(String[] args) throws IOException {
        String aString = "Hello /this is a string";
        Reader inputString = new StringReader(aString);
        BufferedReader br = new BufferedReader(inputString);

        boolean eof = false;
// Parse incoming request
        StreamTokenizer st = new StreamTokenizer(br);

        st.ordinaryChar('/');

        do {
            int token = st.nextToken();

            switch (token) {
                case StreamTokenizer.TT_EOF:
                    System.out.println("End of File encountered.");
                    eof = true;
                    break;

                case StreamTokenizer.TT_EOL:
                    System.out.println("End of Line encountered.");
                    break;

                case StreamTokenizer.TT_WORD:
                    System.out.println("Word: " + st.sval);
                    break;

                case StreamTokenizer.TT_NUMBER:
                    System.out.println("Number: " + st.nval);
                    break;

                default:
                    System.out.println((char) token + " encountered.");

                    if (token == '!') {
                        eof = true;
                    }
            }
        } while (!eof);
    }
}

Found this by Googling "StreamTokenizer forward slash" and got this old link: http://www.dickbaldwin.com/java/Java061.htm