Having trouble with simple tree top grammar

Question

I'm having a play with tree top and I just can't get a simple grammar to work and generate the AST I expect.

My rules are

1: LINE can be made up of one or more PIPED COMMANDs separated by ; 2: a PIPED COMMAND is one or more COMMANDs separated by | 3: a COMMAND is one or more IDENTIFIERS separated by whitespace

I'd expect a tree like this for

hello | abc | def ; abc | test ; cats ;

to generate a tree like this

Line
  PipedCommand
    Command
      Identifier hello
    Command 
      Identifier abc
    Command 
      Identifier def
  PipedCommand
    Command 
      Identifier abc
    Command 
      Identifier test
  PipedCommand
    Command 
      Identifier cats

However I can't get it even just returning piped commands properly, if i specify more than 2, the result is messed up

> test | abc
[Command+Command0 offset=0, "test " (identifier):
  Identifier+Identifier0 offset=0, "test",
 Command+Command0 offset=6, " abc" (identifier):
  Identifier+Identifier0 offset=7, "abc"]
> test | abc | def
[Command+Command0 offset=0, "test " (identifier):
  Identifier+Identifier0 offset=0, "test"]
> 

Grammar currently looks like:

grammar Line
  rule commands
    (command space? '|' commands space?) <Commands> / command
  end

  rule command
    space? identifier space? <Command>
  end

  rule identifier
    [a-zA-Z] [a-zA-Z0-9_]* <Identifier>
  end

  rule space
    [\s]+
  end
end

Hopefully someone can help with a bit!

Thanks

Answer 1

Your parser works on both examples, I just tested it. Try dropping the classes, and just run LineParser.new.parse('some | test') in irb, you'll see the parse tree.

There's something funny in your test program or in your syntax node modules. Perhaps you're feeding an unexpected character into the parser? With the second example output you show, the recursive call to commands must be failing, so it's returning the second option. If so, you must also have set the consume_all_input option to false, or it would fail.

However because it's recursive, it's not going to give you the flat array of Command under your Commands (PipedCommand). You'll get a Commands with the first Command and another Commands, which will contain the other two instances of Command.

If you don't want a nested AST, you should use iteration rather than recursion. This might look like

rule commands
  head:command tail:( '|' command )*
  {
    def ast
      [head] + tail.elements.map(&:command)
    end
  }
end

If that doesn't help you figure it out, please post all the files required to run your example and we'll find the error for you.