Lark: How to ignore whitespace after parsing?

Question

I am creating a REPL for Linux commands.

Since my grammar for command is call: WS? (redirection WS)* argument (WS atom)* WS?, once the parsing is done, I always find whitespace is included as one of the nodes in the parse tree. I understand including WS in the grammar to catch the command line correctly, but I want to filter out them after parsing.

I tried adding %ignore WS at the end of the file, but it didn't work.

Answer 1

You can use a Transformer and have the method for the WS token return Discard.

Transformers make it much easier to convert the result of the parsing into the format that you need for the rest of your program. Since you didn't include your grammar, and your specific use case is too complex to replicate quickly, I'll show an example using the following basic grammar:

GRAMMAR = r"""
?start: ints
ints: (INT WS*)+
%import common (INT, WS)
"""

Before defining a transformer, we can see that all ints and spaces are present in the parsed tree:

>>> Lark(GRAMMAR).parse('12 34 56')
Tree(Token('RULE', 'ints'), [Token('INT', '12'), Token('WS', ' '), Token('INT', '34'), Token('WS', ' '), Token('INT', '56')])

We can define a simple transformer that only transforms WS:

from lark import Lark, Token, Transformer, Discard

class SpaceTransformer(Transformer):
    def WS(self, tok: Token):
        return Discard

Which results in the same tree as before, but now the WS tokens have been removed:

>>> tree = Lark(GRAMMAR).parse('12 34 56')

>>> SpaceTransformer().transform(tree)
Tree(Token('RULE', 'ints'), [Token('INT', '12'), Token('INT', '34'), Token('INT', '56')])

---

The transformer can be expanded further to handle more of the defined tokens:

class SpaceTransformer(Transformer):
    def WS(self, tok: Token):
        return Discard

    def INT(self, tok: Token) -> int:
        return int(tok.value)

That results in the values being proper integers, but they are still in the tree:

>>> tree = Lark(GRAMMAR).parse('12 34 56')

>>> SpaceTransformer().transform(tree)
Tree(Token('RULE', 'ints'), [12, 34, 56])

---

We can take it one step further and define a method for the rule as well - each method in a Transformer that matches a token or rule will automatically be called for each matching parsed value:

class SpaceTransformer(Transformer):
    def WS(self, tok: Token):
        return Discard

    def INT(self, tok: Token) -> int:
        return int(tok.value)

    def ints(self, integers):
        return integers

Now when we transform the tree, we get a list of ints instead of a tree:

>>> tree = Lark(GRAMMAR).parse('12 34 56')

>>> SpaceTransformer().transform(tree)
[12, 34, 56]

While my example used very simple types, you could define a method for your command rule that returns a Command object, or whatever you have defined to represent it. For rules that contain other rules, the outer rules will receive the already transformed objects, just like ints received int objects.

There are also some customizations you can apply to how the transformer methods receive arguments by using the v_args decorator.