How to build a scanner with stringtokenizer

Question

I have a problem in this code, i'm trying to build a scanner for my project in compiler course, the scanner takes any input from the user and separate it into tokens .. the output will be : print each token and it's type(like: number,identifier,keyword,plus sign...etc.) and finally print the number of tokens.

i have tried more inputs and each time the output is identifier, and when i tried to input a number or keyword or + or - the output is identifier..

Here is my code:

import java.util.Scanner;
import java.util.StringTokenizer;

public class MyScanner
{
    public static void main(String[] args)
    {
        String reserved_Keywords[] = { "abstract", "assert", "boolean",
                "break", "byte", "case", "catch", "char", "class", "const",
                "continue", "default", "do", "double", "else", "extends", "false",
                "final", "finally", "float", "for", "goto", "if", "implements",
                "import", "instanceof", "int", "interface", "long", "native",
                "new", "null", "package", "private", "protected", "public",
                "return", "short", "static", "strictfp", "super", "switch",
                "synchronized", "this", "throw", "throws", "transient", "true",
                "try", "void", "volatile", "while" };
        Scanner sc = new Scanner(System.in);
        System.out.println("Enter Your Text: ");
        String str = sc.nextLine();
        StringTokenizer st = new StringTokenizer(str);
        int numofTokens = st.countTokens();
        while( st.hasMoreElements() )
        {
            for (int i = 0; i < reserved_Keywords.length; i++)
            {  
                if ( st.equals(reserved_Keywords[i]) )
                {  
                    System.out.print(st.nextElement() + "\t");
                    System.out.println("Is Reserved Keyword");
                }
            }  

            if ( st.equals("+") )
            {
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Plus Sign");
            }

            else if ( st.equals("-") )
            {
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Minus Sign");
            }

            else if ( st.equals("*") )
            {
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Multiply Sign");
            }

            else if( st.equals("/") )
            {
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Divide Sign");
            }

            else if ( st.equals("=") )
            {
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Assignment Operator");
            }

            else
            {
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Identifier");
            }
        }
        sc.close(); 
        System.out.println("Number of Tokens = " + numofTokens);
    }
}

Answer 1

You always compare (call equals(..)( with the StringTokenizer, not with the Tokens returned by the StringTokenizer.

To lift this, add in the first line of the while Loop

 String TOKEN = st.nextToken();

and then replace all comparisons (calls to equals()) with st instead with TOKEN.

(You should Name the variable of course not with upper case letters, I have done this just for readability)

Then your code will look like:

 StringTokenizer st = new StringTokenizer(str);
    int numofTokens = st.countTokens();
    while( st.hasMoreElements() )
    {   
        String TOKEN = st.nextToken();
        for (int i = 0; i < reserved_Keywords.length; i++)
        {  
            if ( TOKEN.equals(reserved_Keywords[i]) )
            {  
                System.out.print(st.nextElement() + "\t");
                System.out.println("Is Reserved Keyword");
            }
        }  

...