Search code examples
parsingcompiler-errorscompiler-constructionlexical-analysislexical

If lexer can pass the bad token to the parser?


During the lexical analysis phase of compiler if a bad token is encountered then the lexer will go into the error recovery mode and, suppose say, it discards the token until the next semicolon is seen and starts its analysis again. Then overall tokens generated is passed to the parser ?.

What I mean to say if lexer has encountered an error then does the compilation stops at this point or it continues and goes to the parsing phase ?


Solution

  • During the lexical analysis phase of compiler if a bad token is encountered then the lexer will go into the error recovery mode and, suppose say, it discards the token until the next semicolon is seen and starts its analysis again.

    That's only one way of doing it, and not the best.

    Then overall tokens generated is passed to the parser?

    No, only the next legal token.

    What I mean to say if lexer has encountered an error then does the compilation stops at this point or it continues and goes to the parsing phase?

    It continues.

    But for several decades I've been practising the opposite. Instead of having the lexer try to do its own error recovery I just return the offending character to the parser. As the parser is usually equipped with much better error recovery, this leads to a much more fault-tolerant parse.

    Sample lex/flex implementation:

    . return yytext[0];