Parse a list of expressions using pyparsing

Question

I'm trying to use pyparsing to parse simple basic program:

import pyparsing as pp

pp.ParserElement.setDefaultWhitespaceChars(" \t")

EOL = pp.LineEnd().suppress()

# Identifiers is a string + optional $
identifier = pp.Combine(pp.Word(pp.alphas) + pp.Optional("$"))

# Literals (number or double quoted string)
literal = pp.pyparsing_common.number | pp.dblQuotedString

line_number = pp.pyparsing_common.integer

function = pp.Forward()


operand = function | identifier | literal
expression = pp.infixNotation(operand, [
    (pp.oneOf("* / %"), 2, pp.opAssoc.LEFT),
    (pp.oneOf("+ -"), 2, pp.opAssoc.LEFT),
])

assignment = identifier + "=" + expression

# Keywords
PRINT = pp.CaselessKeyword("print")
FOR = pp.CaselessKeyword("for")
TO = pp.CaselessKeyword("to")
STEP = pp.CaselessKeyword("step")
NEXT = pp.CaselessKeyword("next")
CHRS = pp.CaselessKeyword("chr$")

statement = pp.Forward()

print_stmt = PRINT + pp.ZeroOrMore(expression | ";")

for_stmt = FOR + assignment + TO + expression + pp.Optional(STEP + expression)
next_stmt = NEXT

chrs_fn = CHRS + "(" + expression + ")"

function <<= chrs_fn

statement <<= print_stmt | for_stmt | next_stmt | assignment

code_line = pp.Group(line_number + statement + EOL)

program = pp.ZeroOrMore(code_line)

test = """\
10 print 123;
20 print 234; 567;
25 next
30 print 890
"""

print(program.parseString(test).dump())

I do have everything else working except the print clause.

This is the output:

[[10, 'print', 123, ';', 20, 'print', 234, ';', 567, ';', 25, 'next', 30, 'print', 890]]
[0]:
  [10, 'print', 123, ';', 20, 'print', 234, ';', 567, ';', 25, 'next', 30, 'print', 890]

Based on some suggetion I modified my parser but sitll for some reason parser leaks to next row.

How to define list of items for printing properly?

Answer 1

There is a subtle thing going on here, and it bears better documenting when using ParserElement.setDefaultWhitespaceChars. This only updates the whitespace characters for pyparsing expressions created after calling setDefaultWhitespaceChars. The built-in expressions like dblQuotedString and the expressions in pyparsing_common are all defined at import time, and so get the standard set of whitespace characters to skip, which includes '\n'. If you create new copies of them using expr.copy() or just plain expr(), you will get new expressions, which use the updated whitespace characters.

Change:

literal = pp.pyparsing_common.number | pp.dblQuotedString
line_number = pp.pyparsing_common.integer

to:

literal = pp.pyparsing_common.number() | pp.dblQuotedString()
line_number = pp.pyparsing_common.integer()

And I think your leakage issues will be resolved.