Search code examples
pythonantlr4

Antlr4 python runtime error after generating from grammar

I using the following Antlr4 PLSQL grammar files:

https://github.com/antlr/grammars-v4/tree/master/sql/plsql.

From here I downloaded as follows:

wget https://github.com/antlr/grammars-v4/blob/master/sql/plsql/Python3/PlSqlLexerBase.py
wget https://github.com/antlr/grammars-v4/blob/master/sql/plsql/Python3/PlSqlParserBase.py

wget https://github.com/antlr/grammars-v4/blob/master/sql/plsql/PlSqlLexer.g4
wget https://github.com/antlr/grammars-v4/blob/master/sql/plsql/PlSqlParser.g4

mv PlSql*g4 grammars

wget https://www.antlr.org/download/antlr-4.13.1-complete.jar

mv antlr-4.13.1-complete.jar lib

Giving me :

├── lib
│   ├── antlr-4.13.1-complete.jar
├── grammars1
│   ├── PlSqlLexer.g4
│   └── PlSqlParser.g4
├── PlSqlLexer.g4
├── PlSqlParser.py

When I then run:

java -jar ./lib/antlr-4.9.3-complete.jar -Dlanguage=Python3 grammars/*g4

I get the following generated in grammars:

grammars-v4-master  PlSqlLexer.interp  PlSqlParserBase.py  PlSqlParserListener.py  __pycache__
master.zip          PlSqlLexer.py      PlSqlParser.g4      PlSqlParser.py          runPLSQL.py
PlSqlLexer.g4       PlSqlLexer.tokens  PlSqlParser.interp  PlSqlParser.tokens

I then create the runPLSQL.py Python script :

cd grammars
python3 runPLSQL.py ../../plsql/test.sql

But this errored with:

       
  import pandas
Traceback (most recent call last):
  File "/home/me/try2/grammars/runPLSQL.py", line 11, in <module>
    from PlSqlParserListener import PlSqlParserListener
  File "/home/me/try2/grammars/PlSqlParserListener.py", line 6, in <module>
    from PlSqlParser import PlSqlParser
  File "/home/me/try2/grammars/PlSqlParser.py", line 14, in <module>
    from PlSqlParserBase import PlSqlParserBase
  File "/home/me/try2/grammars/PlSqlParserBase.py", line 1, in <module>
    {"payload":{"allShortcutsEnabled":false,"fileTree":{"sql/plsql/Python3":{"items":[{"name":"PlSqlLexerBase.py","path":"sql/plsql/Python3/PlSqlLexerBase.py","contentType":"file"}
NameError: name 'false' is not defined. Did you mean: 'False'?

I had to edit the PlSqlLexerBase.py file as below to overcome this and similar errors:

  1. Replace :false with :False
  2. Replace :true with :True
  3. Replace :null with :None

But now I get this:

  import pandas
Traceback (most recent call last):
  File "/home/me/try2/grammars/runPLSQL.py", line 11, in <module>
    from PlSqlParserListener import PlSqlParserListener
  File "/home/me/try2/grammars/PlSqlParserListener.py", line 6, in <module>
    from PlSqlParser import PlSqlParser
  File "/home/me/try2/grammars/PlSqlParser.py", line 14, in <module>
    from PlSqlParserBase import PlSqlParserBase
ImportError: cannot import name 'PlSqlParserBase' from 'PlSqlParserBase' (/home/me/try2/grammars/PlSqlParserBase.py)

The PlSqlParserBase.py script starts with:

{"payload":{"allShortcutsEnabled":False,"fileTree":{"sql/plsql/Python3":{"items":[{"name":"PlSqlLexerBase.py","path":"sql/plsql/Python3/PlSqlLexerBase.py","contentType":"file"},{"name":"PlSqlParserBase.py","path":"sql/plsql/Python3/PlSqlParserBase.py","contentType":"file"}],"totalCount":2},"sql/plsql":{"items":[{"name":"CSharp","path":"sql/plsql/CSharp","contentTy......

I notice it references relative pathnames, should all the paths/files exist?

The top of the runPLSQL.py script is:

import os

import pandas
from antlr4 import InputStream, ParseTreeWalker
from antlr4.CommonTokenStream import CommonTokenStream
from pandas import DataFrame

#from PlSql.grammar.PlSqlListener import PlSqlListener
#from PlSql.grammar.PlSqlLexer import PlSqlLexer
#from PlSql.grammar.PlSqlParser import PlSqlParser
from PlSqlParserListener import PlSqlParserListener
from PlSqlLexer import PlSqlLexer
from PlSqlParser import PlSqlParser


from tabulate import tabulate


class SQLParser(PlSqlListener):

I also looked in the autogenerated master.zip and it contains file names being referneced, though not the exact same path:

unzip -l master.zip | grep PlSql
      896  2024-02-20 11:13   grammars-v4-master/sql/plsql/CSharp/PlSqlLexerBase.cs
      720  2024-02-20 11:13   grammars-v4-master/sql/plsql/CSharp/PlSqlParserBase.cs
      367  2024-02-20 11:13   grammars-v4-master/sql/plsql/Cpp/PlSqlLexerBase.h
      614  2024-02-20 11:13   grammars-v4-master/sql/plsql/Cpp/PlSqlParserBase.h
   109954  2024-02-20 11:13   grammars-v4-master/sql/plsql/Dart/PlSqlLexer.g4
      358  2024-02-20 11:13   grammars-v4-master/sql/plsql/Dart/PlSqlLexerBase.dart
   149085  2024-02-20 11:13   grammars-v4-master/sql/plsql/Dart/PlSqlParser.g4
      571  2024-02-20 11:13   grammars-v4-master/sql/plsql/Dart/PlSqlParserBase.dart
      354  2024-02-20 11:13   grammars-v4-master/sql/plsql/Java/PlSqlLexerBase.java
      610  2024-02-20 11:13   grammars-v4-master/sql/plsql/Java/PlSqlParserBase.java
      259  2024-02-20 11:13   grammars-v4-master/sql/plsql/JavaScript/PlSqlLexerBase.js
      506  2024-02-20 11:13   grammars-v4-master/sql/plsql/JavaScript/PlSqlParserBase.js
   115277  2024-02-20 11:13   grammars-v4-master/sql/plsql/PlSqlLexer.g4
   228726  2024-02-20 11:13   grammars-v4-master/sql/plsql/PlSqlParser.g4
      157  2024-02-20 11:13   grammars-v4-master/sql/plsql/Python3/PlSqlLexerBase.py
      364  2024-02-20 11:13   grammars-v4-master/sql/plsql/Python3/PlSqlParserBase.py
      334  2024-02-20 11:13   grammars-v4-master/sql/plsql/TypeScript/PlSqlLexerBase.ts
      604  2024-02-20 11:13   grammars-v4-master/sql/plsql/TypeScript/PlSqlParserBase.ts
Solution

  • Check the files you downloaded for

    https://github.com/antlr/grammars-v4/blob/master/sql/plsql/Python3/PlSqlLexerBase.py

    and

    wget https://github.com/antlr/grammars-v4/blob/master/sql/plsql/Python3/PlSqlParserBase.py

    They should be very small Python scripts, but you seem to have somehow got the github JSON file list. Maybe your wget command needs looking at.