I am feeling really dumb right now but I am not understanding the concept of tokens. I am currently reading Dr. Stroustrup's textbook, Programming Principals and Practices, am and in Chapter 6 learning about tokens and writing programs in C++. I have gotten every concept in the book thus far, but this has really got me stuck. Can anyone please dumb it down for me? Thank you!
I have honestly tried to watch YouTube videos that have not been of any help along with reading the textbook that hasn't helped either.
A token corresponds roughly to a word of source-code. For example, in a line of source code like this:
int a = 10;
The tokens would be "int", "a", "=", "10", and ";".
In a compiler, it's the job of the tokenizer to interpret the source code files (which are really just plain-text files, i.e. a series of characters) into a series of tokens (which have some sort of specific meaning in the context of the language). The series of tokens will then be fed into the next stage of the compiler for further processing.