Search code examples
ctokenlexical-analysiscompiler-construction

Number of tokens in code


 void main()
 {
    i/* nt */a=10;
    return;
 }

Number of tokens in above code:?

I have calculated it as 13 by removing comment and not considering white space

Does we consider 'i' and 'a' as single token after removing comment? Thanks lot


Solution

  • It's two tokens. A comment is replaced with a single space, so after the comment is removed it's equivalent to:

    i a=10;
    

    The space separates tokens.

    From Wikipedia article on the C Preprocessor

    1. Tokenization: The preprocessor breaks the result into preprocessing tokens and whitespace. It replaces comments with whitespace.