parsingscriptingprogramming-languagesantlrgrammar
Why isn't this expression being parsed correctly?
NOTE: This is a continuation of the topic posted HERE.
I'm working on a parser for the Jass scripting language (here's an excellent API reference for it) so that I may use it as in interpreter for another language. Using ANTLR4 + ANTLRWorks 2, I have run this complex script to test the lexer/parser's strength, and have passed nearly all tests. The part where it fails is on in an 'elseif' statement, containing an expression with:
- an outer parenthesis...
- an array element...
- a boolean/binary operation, AND...
- a unary constant integer
...like so:
elseif(si__DroneSystem___data_V[this]!=-1)then (line #53 of the script).
Any changes I've made to the grammar fails to get ANTLR to recognize this input as a proper expression. The following grammar is what I've managed to write, thus far:
grammar Jass;
//----------------------------------------------------------------------
// Global Declarations
//----------------------------------------------------------------------
program : file+
;
file : declaration* function
;
declaration : globals | typedef | native_func
;
typedef : KEYWORD_TYPE identifier KEYWORD_EXTENDS (TYPE_HANDLE | identifier)
;
globals : KEYWORD_GLOBALS global_var_list KEYWORD_ENDGLOBALS
;
global_var_list : var_declaration*
;
native_func : KEYWORD_CONSTANT? KEYWORD_NATIVE func_declaration
;
func_declaration : identifier KEYWORD_TAKES (KEYWORD_NOTHING | parameter_list) KEYWORD_RETURNS (KEYWORD_NOTHING | type)
;
parameter_list : type identifier (',' type identifier)*
;
function : KEYWORD_CONSTANT? KEYWORD_FUNCTION func_declaration local_var_list statement_list KEYWORD_ENDFUNCTION
;
//----------------------------------------------------------------------
// Local Declarations
//----------------------------------------------------------------------
local_var_list : (KEYWORD_LOCAL? var_declaration)*
;
var_declaration : KEYWORD_CONSTANT type identifier '=' expression
| type identifier ('=' expression)? | type TYPE_ARRAY identifier
;
//----------------------------------------------------------------------
// Statements
//----------------------------------------------------------------------
statement_list : statement*
;
statement : set | call | if_statement | loop | exitwhen | return_statement | debug
;
set : KEYWORD_SET identifier '=' expression | KEYWORD_SET identifier OPENBRACKET expression CLOSEBRACKET '=' expression
;
call : KEYWORD_CALL identifier OPENPARENTHESIS args? CLOSEPARENTHESIS
;
args : expression (COMMA expression)*
;
if_statement : KEYWORD_IF expression KEYWORD_THEN statement_list else_clause? KEYWORD_ENDIF
;
else_clause : KEYWORD_ELSEIF ((OPENPARENTHESIS expression CLOSEPARENTHESIS) | expression) KEYWORD_THEN statement_list
| KEYWORD_ELSE ((OPENPARENTHESIS statement_list CLOSEPARENTHESIS) | statement_list) else_clause?
;
loop : KEYWORD_LOOP statement_list KEYWORD_ENDLOOP
;
// must appear in a loop
exitwhen : KEYWORD_EXITWHEN expression
;
return_statement : KEYWORD_RETURN expression?
;
debug : KEYWORD_DEBUG (set | call | if_statement | loop)
;
//----------------------------------------------------------------------
// Expressions
//----------------------------------------------------------------------
expression : parenthesis
| func_call
| array_ref
| (boolean_expression | binary_operation)
| unary_operation
| function_reference
| const_statement
| identifier
;
binary_operation : terminal (('+'|'-'|'*'|'/'|'>'|'<'|'=='|'!='|'>='|'<=') terminal)
;
unary_operation : ('+'|'-'|'not') terminal
;
boolean_expression : ('and'|'not')? terminal (('=='|'!=') terminal) ('and'|'or')?
;
terminal : factor*/(factor)
;
factor : identifier
| const_statement
| parenthesis
| brackets
;
parenthesis : OPENPARENTHESIS expression CLOSEPARENTHESIS
;
brackets : OPENBRACKET expression CLOSEBRACKET
;
// expression must be integer or real when used with unary '+'
func_call : identifier OPENPARENTHESIS args? CLOSEPARENTHESIS
;
array_ref : identifier OPENBRACKET expression CLOSEBRACKET
;
function_reference : KEYWORD_FUNCTION identifier
;
const_statement : INTEGER_CONST | REAL_CONST | BOOL_CONST | STRING_CONST | ASSIGNMENT_TYPE_NULL
;
FOURCC : QUOTATION_SINGLE . . . . QUOTATION_SINGLE
;
INTEGER_CONST : DECIMAL | OCTAL | HEXIDECIMAL | FOURCC
;
DECIMAL : (DIGIT)+ | (DIGIT+) '.' (DIGIT+)?
;
OCTAL : '0'..'7'+
;
HEXIDECIMAL : '$'(DIGIT|'a'..'f'|'A'..'F')+ | '0'('x'|'X')(DIGIT|'a'..'f'|'A'..'F')+
;
REAL_CONST : (DIGIT)+'.'(DIGIT)* | '.'(DIGIT)+
;
BOOL_CONST : ASSIGNMENT_TYPE_TRUE | ASSIGNMENT_TYPE_FALSE
;
// any double-quotes in the string must be escaped with \
STRING_CONST : QUOTATION_DOUBLE .*? QUOTATION_DOUBLE
;
//----------------------------------------------------------------------
// Base
//----------------------------------------------------------------------
type : nativetype | commontype
;
identifier : ID
;
//////////////////////////////////////////////////////////////////////////////////////////////
// TYPES
//////////////////////////////////////////////////////////////////////////////////////////////
TYPE_BOOLEAN : 'boolean'
;
TYPE_CODE : 'code'
;
TYPE_HANDLE : 'handle'
;
TYPE_INTEGER : 'integer'
;
TYPE_REAL : 'real'
;
TYPE_STRING : 'string'
;
TYPE_ARRAY : 'array'
;
nativetype : TYPE_BOOLEAN
| TYPE_CODE
| TYPE_HANDLE
| TYPE_INTEGER
| TYPE_REAL
| TYPE_STRING
| TYPE_ARRAY
;
TYPE_ABILITY : 'ability'
;
TYPE_AGENT : 'agent'
;
TYPE_AIDIFFICULTY : 'aidifficulty'
;
TYPE_ALLIANCETYPE : 'alliancetype'
;
TYPE_ATTACKTYPE : 'attacktype'
;
TYPE_BLENDMODE : 'blendmode'
;
TYPE_BOOLEXPR : 'boolexpr'
;
TYPE_BUFF : 'buff'
;
TYPE_BUTTON : 'button'
;
TYPE_CAMERAFIELD : 'camerafield'
;
TYPE_CAMERASETUP : 'camerasetup'
;
TYPE_CONDITIONFUNC : 'conditionfunc'
;
TYPE_DAMAGETYPE : 'damagetype'
;
TYPE_DEFEATCONDITION : 'defeatcondition'
;
TYPE_DESTRUCTABLE : 'destructable'
;
TYPE_DIALOG : 'dialog'
;
TYPE_DIALOGEVENT : 'dialogevent'
;
TYPE_EFFECT : 'effect'
;
TYPE_EVENTID : 'eventid'
;
TYPE_FGAMESTATE : 'fgamestate'
;
TYPE_FILTERFUNC : 'filterfunc'
;
TYPE_FOGMODIFIER : 'fogmodifier'
;
TYPE_FOGSTATE : 'fogstate'
;
TYPE_FORCE : 'force'
;
TYPE_GAMECACHE : 'gamecache'
;
TYPE_GAMEDIFFICULTY : 'gamedifficulty'
;
TYPE_GAMEEVENT : 'gameevent'
;
TYPE_GAMESPEED : 'gamespeed'
;
TYPE_GAMESTATE : 'gamestate'
;
TYPE_GAMETYPE : 'gametype'
;
TYPE_GROUP : 'group'
;
TYPE_HASHTABLE : 'hashtable'
;
TYPE_IGAMESTATE : 'igamestate'
;
TYPE_IMAGE : 'image'
;
TYPE_ITEM : 'item'
;
TYPE_ITEMPOOL : 'itempool'
;
TYPE_ITEMTYPE : 'itemtype'
;
TYPE_LEADERBOARD : 'leaderboard'
;
TYPE_LIGHTNING : 'lightning'
;
TYPE_LIMITOP : 'limitop'
;
TYPE_LOCATION : 'location'
;
TYPE_MAPCONTROL : 'mapcontrol'
;
TYPE_MAPDENSITY : 'mapdensity'
;
TYPE_MAPFLAG : 'mapflag'
;
TYPE_MAPSETTING : 'mapsettings'
;
TYPE_MAPVISIBILITY : 'mapvisibility'
;
TYPE_MULTIBOARD : 'multiboard'
;
TYPE_MULTIBOARDITEM : 'multiboarditem'
;
TYPE_PATHINGTYPE : 'pathingtype'
;
TYPE_PLACEMENT : 'placement'
;
TYPE_PLAYER : 'player'
;
TYPE_PLAYERCOLOR : 'playercolor'
;
TYPE_PLAYEREVENT : 'playerevent'
;
TYPE_PLAYERGAMERESULT : 'playergameresult'
;
TYPE_PLAYERSCORE : 'playerscore'
;
TYPE_PLAYERSLOTSTATE : 'playerslotstate'
;
TYPE_PLAYERSTATE : 'playerstate'
;
TYPE_PLAYERUNITEVENT : 'playerunitevent'
;
TYPE_QUEST : 'quest'
;
TYPE_QUESTITEM : 'questitem'
;
TYPE_RACE : 'race'
;
TYPE_RACEPREFERENCE : 'racepreference'
;
TYPE_RARITYCONTROL : 'raritycontrol'
;
TYPE_RECT : 'rect'
;
TYPE_REGION : 'region'
;
TYPE_SOUND : 'sound'
;
TYPE_SOUNDTYPE : 'soundtype'
;
TYPE_STARTLOCPRIO : 'startlocprio'
;
TYPE_TERRAINDEFORMATION : 'terraindeformation'
;
TYPE_TEXMAPFLAGS : 'texmapflags'
;
TYPE_TEXTTAG : 'texttag'
;
TYPE_TIMER : 'timer'
;
TYPE_TIMERDIALOG : 'timerdialog'
;
TYPE_TRACKABLE : 'trackable'
;
TYPE_TRIGGER : 'trigger'
;
TYPE_TRIGGERACTION : 'triggeraction'
;
TYPE_TRIGGERCONDITION : 'triggercondition'
;
TYPE_UBERSPLAT : 'ubersplat'
;
TYPE_UNIT : 'unit'
;
TYPE_UNITEVENT : 'unitevent'
;
TYPE_UNITPOOL : 'unitpool'
;
TYPE_UNITSTATE : 'unitstate'
;
TYPE_UNITTYPE : 'unittype'
;
TYPE_VERSION : 'version'
;
TYPE_VOLUMEGROUP : 'volumegroup'
;
TYPE_WEAPONTYPE : 'weapontype'
;
TYPE_WEATHEREFFECT : 'weathereffect'
;
TYPE_WIDGET : 'widget'
;
TYPE_WIDGETEVENT : 'widgetevent'
;
commontype : TYPE_ABILITY
| TYPE_AGENT
| TYPE_AIDIFFICULTY
| TYPE_ALLIANCETYPE
| TYPE_ATTACKTYPE
| TYPE_BLENDMODE
| TYPE_BOOLEXPR
| TYPE_BUFF
| TYPE_BUTTON
| TYPE_CAMERAFIELD
| TYPE_CAMERASETUP
| TYPE_CONDITIONFUNC
| TYPE_DAMAGETYPE
| TYPE_DEFEATCONDITION
| TYPE_DESTRUCTABLE
| TYPE_DIALOG
| TYPE_DIALOGEVENT
| TYPE_EFFECT
| TYPE_EVENTID
| TYPE_FGAMESTATE
| TYPE_FILTERFUNC
| TYPE_FOGMODIFIER
| TYPE_FOGSTATE
| TYPE_FORCE
| TYPE_GAMECACHE
| TYPE_GAMEDIFFICULTY
| TYPE_GAMEEVENT
| TYPE_GAMESPEED
| TYPE_GAMESTATE
| TYPE_GAMETYPE
| TYPE_GROUP
| TYPE_HASHTABLE
| TYPE_IGAMESTATE
| TYPE_IMAGE
| TYPE_ITEM
| TYPE_ITEMPOOL
| TYPE_ITEMTYPE
| TYPE_LEADERBOARD
| TYPE_LIGHTNING
| TYPE_LIMITOP
| TYPE_LOCATION
| TYPE_MAPCONTROL
| TYPE_MAPDENSITY
| TYPE_MAPFLAG
| TYPE_MAPSETTING
| TYPE_MAPVISIBILITY
| TYPE_MULTIBOARD
| TYPE_MULTIBOARDITEM
| TYPE_PATHINGTYPE
| TYPE_PLACEMENT
| TYPE_PLAYER
| TYPE_PLAYERCOLOR
| TYPE_PLAYEREVENT
| TYPE_PLAYERGAMERESULT
| TYPE_PLAYERSCORE
| TYPE_PLAYERSLOTSTATE
| TYPE_PLAYERSTATE
| TYPE_PLAYERUNITEVENT
| TYPE_QUEST
| TYPE_QUESTITEM
| TYPE_RACE
| TYPE_RACEPREFERENCE
| TYPE_RARITYCONTROL
| TYPE_RECT
| TYPE_REGION
| TYPE_SOUND
| TYPE_SOUNDTYPE
| TYPE_STARTLOCPRIO
| TYPE_TERRAINDEFORMATION
| TYPE_TEXMAPFLAGS
| TYPE_TEXTTAG
| TYPE_TIMER
| TYPE_TIMERDIALOG
| TYPE_TRACKABLE
| TYPE_TRIGGER
| TYPE_TRIGGERACTION
| TYPE_TRIGGERCONDITION
| TYPE_UBERSPLAT
| TYPE_UNIT
| TYPE_UNITEVENT
| TYPE_UNITPOOL
| TYPE_UNITSTATE
| TYPE_UNITTYPE
| TYPE_VERSION
| TYPE_VOLUMEGROUP
| TYPE_WEAPONTYPE
| TYPE_WEATHEREFFECT
| TYPE_WIDGET
| TYPE_WIDGETEVENT
;
//////////////////////////////////////////////////////////////////////////////////////////////
ASSIGNMENT_TYPE_NULL : 'null'
;
ASSIGNMENT_TYPE_INTEGER : DIGIT
;
ASSIGNMENT_TYPE_REAL : REAL_CONST
;
ASSIGNMENT_TYPE_TRUE : 'true'
;
ASSIGNMENT_TYPE_FALSE : 'false'
;
KEYWORD_DEBUG : 'debug'
;
KEYWORD_EXTENDS : 'extends'
;
KEYWORD_NATIVE : 'native'
;
KEYWORD_FUNCTION : 'function'
;
KEYWORD_ENDFUNCTION : 'endfunction'
;
KEYWORD_TAKES : 'takes'
;
KEYWORD_NOTHING : 'nothing'
;
KEYWORD_RETURNS : 'returns'
;
KEYWORD_CALL : 'call'
;
KEYWORD_RETURN : 'return'
;
KEYWORD_GLOBALS : 'globals'
;
KEYWORD_ENDGLOBALS : 'endglobals'
;
KEYWORD_LOCAL : 'local'
;
KEYWORD_CONSTANT : 'constant'
;
KEYWORD_SET : 'set'
;
KEYWORD_IF : 'if'
;
KEYWORD_THEN : 'then'
;
KEYWORD_ELSEIF : 'elseif'
;
KEYWORD_ELSE : 'else'
;
KEYWORD_ENDIF : 'endif'
;
KEYWORD_LOOP : 'loop'
;
KEYWORD_EXITWHEN : 'exitwhen'
;
KEYWORD_ENDLOOP : 'endloop'
;
KEYWORD_TYPE : 'type'
;
ID : (LETTER)((LETTER|DIGIT|'_'+)*)?
;
fragment
LETTER : '\u0024' // $
| '\u0041'..'\u005a' // A-Z
| '\u005f' // _
| '\u0061'..'\u007a' // a-z
| '\u00c0'..'\u00d6' // Latin Capital Letter A with grave - Latin Capital letter O with diaeresis
| '\u00d8'..'\u00f6' // Latin Capital letter O with stroke - Latin Small Letter O with diaeresis
| '\u00f8'..'\u00ff' // Latin Small Letter O with stroke - Latin Small Letter Y with diaeresis
| '\u0100'..'\u1fff' // Latin Capital Letter A with macron - Latin Small Letter O with stroke and acute
| '\u3040'..'\u318f' // Hiragana
| '\u3300'..'\u337f' // CJK compatibility
| '\u3400'..'\u3d2d' // CJK compatibility
| '\u4e00'..'\u9fff' // CJK compatibility
| '\uf900'..'\ufaff' // CJK compatibility
;
fragment
DIGIT : '0'..'9'/*'\u0030'..'\u0039' // 0-9
| '\u0660'..'\u0669' // Arabic-Indic Digit 0-9
| '\u06f0'..'\u06f9' // Extended Arabic-Indic Digit 0-9
| '\u0966'..'\u096f' // Devanagari 0-9
| '\u09e6'..'\u09ef' // Bengali 0-9
| '\u0a66'..'\u0a6f' // Gurmukhi 0-9
| '\u0ae6'..'\u0aef' // Gujarati 0-9
| '\u0b66'..'\u0b6f' // Oriya 0-9
| '\u0be7'..'\u0bef' // Tami 0-9
| '\u0c66'..'\u0c6f' // Telugu 0-9
| '\u0ce6'..'\u0cef' // Kannada 0-9
| '\u0d66'..'\u0d6f' // Malayala 0-9
| '\u0e50'..'\u0e59' // Thai 0-9
| '\u0ed0'..'\u0ed9' // Lao 0-9
| '\u1040'..'\u1049' // Myanmar 0-9?*/
;
OPENPARENTHESIS : '('
;
CLOSEPARENTHESIS : ')'
;
OPENBRACKET : '['
;
CLOSEBRACKET : ']'
;
QUOTATION_DOUBLE : '"'
;
QUOTATION_SINGLE : '\''
;
COMMA : ','
;
WS : (' ' | '\t' | '\n'+)+ {skip();}
;
LINE_COMMENT : '//' ~[\r\n]* -> channel(HIDDEN)
;
And from ANTLRWorks...
THIS file is the output log from using TestRig (starting with the first error), and here is an image of the generated parse tree where the first error occurs:
CLICK HERE to enlarge
TO ANYONE who can help me fix this issue: I will gladly upvote your answers, as well as your next 3 questions if you are marked as the answer to this question.
Thanks!
Solution
When looking at the BNF rules of an if statement:
ifthenelse
::= 'if' expr 'then' newline statement_list else_clause? 'endif'
else_clause
::= 'else' newline statement_list
| 'elseif' expr 'then' newline statement_list else_clause?
your translation:
if_statement : KEYWORD_IF expression KEYWORD_THEN statement_list else_clause? KEYWORD_ENDIF
;
else_clause : KEYWORD_ELSEIF ((OPENPARENTHESIS expression CLOSEPARENTHESIS) | expression) KEYWORD_THEN statement_list
| KEYWORD_ELSE ((OPENPARENTHESIS statement_list CLOSEPARENTHESIS) | statement_list) else_clause?
;
is incorrect (you have an optional else_clause in the KEYWORD_ELSE alternative).
It should be:
if_statement : KEYWORD_IF expression KEYWORD_THEN statement_list else_clause? KEYWORD_ENDIF
;
else_clause : KEYWORD_ELSE statement_list
| KEYWORD_ELSEIF expression KEYWORD_THEN statement_list else_clause?
;
And not that you don't need ((OPENPARENTHESIS expression CLOSEPARENTHESIS) | expression) since a expression already matches '(' expression ')'.
But the observations above are not the cause of your problem(s). The real issue is that your grammar does not account for unary expressions. It does not match the -1 in the expression si__DroneSystem___data_V[this]!=-1.
Change your expression rule into this:
expression : OPENPARENTHESIS expression CLOSEPARENTHESIS
| OPENBRACKET expression CLOSEBRACKET
| func_call
| array_ref
| function_reference
| const_statement
| identifier
| '+' expression
| '-' expression
| 'not' expression
| expression ('*'|'/') expression
| expression ('+'|'-') expression
| expression ('>'|'<'|'=='|'!='|'>='|'<=') expression
| expression ('and'|'or') expression
| identifier
;
Now input like this:
if this==null then
return
elseif(si__DroneSystem___data_V[this]!=-1)then
return
endif
will be parsed as follows:
- Best way to parse string of email addresses
- ANTLR4 - Token recognition error and mismatched input
- Is there a Python parsing library that can parse a TOML-like format that specifies nested fields with [ParentHeader_ChildSection]?
- Convert Month String to Integer in Java
- Converting a hexadecimal string to a decimal integer
- Ugly Triple Indirection: Extensible Buffer Management Struct
- Does anyone have an easy solution to parsing Exp-Golomb codes using C++?
- Python/Json:Expecting property name enclosed in double quotes
- Fast string parsing in C#
- Parsing variable length strings of fixed column widths
- How can I parse text from a *.PG type file?
- How to reformat JSON in Notepad++
- How to do an Integer.parseInt() for a decimal number?
- Read fixed width record from text file
- How can I read and parse CSV files in C++?
- Problems while transforming a python dict to a list of triples?
- BASH - read lines from section of file
- How can a Python program load and read specific lines from a file?
- Reading Serial Port data and saving to .csv Python
- Parsing with a space (sscanf or strtok)
- How to get StyleRules as a string output using ExCSS Parser
- Find out a work around to bypass Java.time bug with spelling of name of month
- How to parse TOML document with Java
- Fatal error - Parse error: syntax error, unexpected ':', expecting - Wordpress
- Neovim command to create print statement for Python function arguments
- java.io.StreamTokenizer.TT_NUMBER: Floating or Integer?
- How can I get the LineNumber of the element when using Jsoup?
- How to parse a string into a DateTime object in Perl?
- How can I parse (read) and use JSON in Python?
- Getting the gold price from the web - python