How can I debug/trace/dump the paths the ANTLR4-generated lexer chooses? Many times I am out of luck to see in the grammar itself what went wrong.
You can use ANTLRWorks. Under run -> Interpret lexer you can see the output for the lexer grammar. Maybe also click Window -> Lexer debugger Controller window, if it's not automatically displayed. Under it, you can see types, tokens, channels, modes, lookahead. Tokens are quite useful to see which "string" was recognized as which token. Also, if I have lexer modes I always use antlrworks.