Debugging Python ANTLR4 Grammar

Question

I'm having an issue with my ANTLR4 grammar not parsing a string correctly. I'm more interested in learning how to solve my problem than solving my specific problem. How can I generate any type of debug information? I want to know what the parser is "thinking" as it parses the string.

The grammar can be found here: https://github.com/Metrink/metrink-fe/blob/master/metrink.g4

I'm using the simple test string: -1d metric('blah', 'blah', 'blah')

I get the following error: 1:2 missing TIME_INDICATOR at 'd'

The grammar defines TIME_INDICATOR as [shmd] so I'm not sure how it's missing a TIME_INDICATOR at the character d when that is one of the possible tokens. What am I missing here?

I'm using Python3 generated from ANTLR4.

Answer 1

What I usually do is first dump the tokens to see if the actual tokens the parser expects are created.

You can do that with a small test class like this (easily ported to Python):

public class Main {

    static void test(String input) {
        
        metrinkLexer lexer = new metrinkLexer(new ANTLRInputStream(input));
        CommonTokenStream tokenStream = new CommonTokenStream(lexer);
        tokenStream.fill();

        System.out.printf("input: `%s`\n", input);

        for (Token token : tokenStream.getTokens()) {
            if (token.getType() != TLexer.EOF) {
                System.out.printf("  %-20s %s\n", metrinkLexer.VOCABULARY.getSymbolicName(token.getType()), token.getText());
            }
        }

        System.out.println();
    }

    public static void main(String[] args) throws Exception {
        test("-1d metric('blah', 'blah', 'blah')");
    }
}

If you run the code above, the following will get printed to your console:

input: `-1d metric('blah', 'blah', 'blah')`
  MINUS                -
  INTEGER_LITERAL      1
  IDENTIFIER           d
  METRIC               metric
  LPAREN               (
  STRING_LITERAL       'blah'
  COMMA                ,
  STRING_LITERAL       'blah'
  COMMA                ,
  STRING_LITERAL       'blah'
  RPAREN               )

As you can see, the d is being tokenized as a IDENTIFIER instead of an TIME_INDICATOR. This is because the IDENTIFIER rule is defined before your TIME_INDICATOR rule. The lexer does not "listen" to what the parser might need, it simply matches the most characters as possible, and if two or more rules match the same amount of characters, the rule defined first "wins".

So, d can either be tokenized as TIME_INDICATOR or an IDENTIFIER. If this is dependent on context, I suggest you tokenize it as a IDENTIFIER (and remove TIME_INDICATOR) and create a parser rule like this:

relative_time_literal:
    MINUS? INTEGER_LITERAL time_indicator;

time_indicator:
    {_input.LT(1).getText().matches("[shmd]")}? IDENTIFIER;

The { ... }? is called a predicate: Semantic predicates in ANTLR4?

Also, FALSE and TRUE will need to be placed before the IDENTIFIER rule.

EDIT April 6 2024

Petr Pivonka wrote:

PAY ATTENTION! The note "easily ported to Python" needs to be explained! [...]

In Python that could look like this:

import antlr4
from metrinkLexer import metrinkLexer


def test(source):
    lexer = metrinkLexer(antlr4.InputStream(source))
    token_stream = antlr4.CommonTokenStream(lexer)
    token_stream.fill()

    print(f"input: {source}")

    for token in [t for t in token_stream.tokens if t.type != -1]:
        print(f"  {lexer.symbolicNames[token.type].ljust(20)}{token.text}")


if __name__ == '__main__':
    test("-1d metric('blah', 'blah', 'blah')")

which will print:

input: -1d metric('blah', 'blah', 'blah')
  MINUS               -
  INTEGER_LITERAL     1
  IDENTIFIER          d
  METRIC              metric
  LPAREN              (
  STRING_LITERAL      'blah'
  COMMA               ,
  STRING_LITERAL      'blah'
  COMMA               ,
  STRING_LITERAL      'blah'
  RPAREN              )

In other words:

• metrinkLexer.VOCABULARY.getSymbolicName(type) becomes metrinkLexer.symbolicNames[type]
• token.getType() becomes token.type
• token.getText() becomes token.text

Petr Pivonka wrote:

It took me one day of labouring before I found that Java and Python are completely different nad that Python one is just empty shell.

That is not true. Everything in the Java API is also in the Python API (and also in C#, JavaScript, TypeScript, etc).