Search code examples
javastringantlrliteralsantlrworks

Antlr error when typing string

When I type a String in my Java editor using my Antlr Defined Language I get an error as soon as i type the first quotation(").

input:

    StatusString = ";
    break;

error:

java.lang.IllegalStateException: Lexer com.lexer.IpIdeLexer@5ab38d6b
  returned null token but lexerInput.readLength()=21
  lexer-state: null
  tokenStartOffset=557, readOffset=578, lookaheadOffset=579
  Chars: "";\n        break;\n}\n\n" - these characters need to be tokenized.
Fix the lexer to not return null token in this state.

I suspect it is because of how i defined my String Literal in my grammer.

Grammar Definition

STRING_LITERAL
    :  '"' ( EscapeSequence | ~('\\'|'"') )* '"' {setText(getText().substring(1,    getText().length()-1));}
    ;

fragment
EscapeSequence
    :   '\\' ('b'|'t'|'n'|'f'|'r'|'\"'|'\''|'\\')
    |   OctalEscape
    ;

I suspect what is happening is that as soon as it sees the first quotation it is trying to extract the string. This is done real time because its an IDE that is using the lexer to color code the syntax. Is there something I can do with my grammar to prevent this error?

Edit: What I was thinking of doing was have it check to the end of the line. The IDE would just color the line orange starting from the quote until the quote is completed by the user, like netbeans or eclipse does. Im just not sure how to go about this through the grammar so its reflected in the lexer/parser

Solution

  • If you're going to try and prevent the error, then you would also miss the error if you really forget a quote somewhere, not only when you've just typed a starting quote.

    I'd solve this by letting your IDE wait a couple of milliseconds (1/2 second or so) so that you have some time to enter a closing quote, or perhaps let the IDE insert a closing quote automatically?

    Trying to adjust the grammar so that it also allows for invalid input is not a good idea, IMO.