StreamTokenizer Behavior for division operator "/"

Question

Take a look at the demo code for StreamTokenizer here. It doesn't seems to work properly when there is / in string(Just add / in between string in StringReader). Here is the code from mentioned link,

StreamTokenizer tokenizer = new StreamTokenizer(
    new StringReader("Mary had 1 little lamb..."));

    while(tokenizer.nextToken() != StreamTokenizer.TT_EOF){

        if(tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println(tokenizer.sval);
        } else if(tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println(tokenizer.nval);
        } else if(tokenizer.ttype == StreamTokenizer.TT_EOL) {
            System.out.println();
        }
    }

For example, for string "Mary had 1 little lamb...", output is

Mary
had
1.0
little
lamb...

For string, "Mary had 1 /little lamb...", output is

Mary
had
1.0

• Does / work as EOF token? If so, why?
• Is there any way to distinguish / as a different token other than EOF.

Answer 1

As per the documentation, / is the comment character in a StreamTokenizer. Everything that comes after it, until an EOL or EOF will be ignored and won't be tokenized.

E.g., to continue the example you've given, if the string is "Mary had 1 / 2\n little lamb...", 2 is commented out and won't be tokenized, and the tokenization will resume after the \n. So the output would be:

Mary
had
1.0
little
lamb...