I'm working on a language interpreter for a programming language I made up. Here's some example code, which should work but currently dies with Syntax error at offset 45.
when reading this testcase.
{
foo = { "min" : 1 ,"max" : 5};
foo["min"]
}
The correct interpretation is for the first line with foo to create a map and store it in a variable named foo, the second line looks up the value of the field min in the record foo, and the starting/ending curlies together with the semicolon wrap the two expressions into an expr_seq
(i.e. a block) which evaluates to the same thing as the last expr
in it.
A simplified version of my parser.mly is as follows:
%token <int> INT
%token <string> VAR
%token SEMI COMMA COLON ASSIGN QUOTE
%token LBRACK RBRACK LCURL RCURL
%token EOF
%start <int> main
%%
main:
| EOF
{ failwith "empty input" }
| e = exp EOF
{ e }
exp:
| INT
{ 0 }
| e = exp LBRACK v = q_var RBRACK
{ (* map lookup *) 0 }
| v = VAR ASSIGN e = exp
{ (* assign to var *) 0 }
| v = VAR LBRACK f = q_var RBRACK ASSIGN e = exp
{ (* assign to map field *) 0 }
| v = VAR
{ Printf.printf "lookup %s\n" v; 0 }
| LCURL e = expr_seq RCURL
{ (* Block expression *) 0 }
| LCURL f = fields RCURL
{ (* map literal *)0 }
fields:
| v = q_var COLON e = exp
{ [(v, e)] }
| v = q_var COLON e = exp COMMA vt = fields
{ (v, e) :: vt }
q_var:
| QUOTE v = VAR QUOTE
{ Printf.printf "qvar %s\n" v; v }
expr_seq:
| e = exp
{[e]}
|e1 = exp SEMI e2 = expr_seq
{e1 :: e2}
Trying to debug it on my own, I found that if you removed the following | v = VAR LBRACK f = q_var RBRACK ASSIGN e = exp
it will parse it and run correctly, but I'd really like to be able to set things in maps.
I'm 98% confident that the problem lies in my mly file, but a simplified version of my lexer.mll is as follows:
{
open Parser
open Printf
}
rule token = parse
| [' ' '\t' '\n']
{ token lexbuf }
| "="
{ASSIGN}
| ['1'-'9']['0'-'9']* as i
{ INT (int_of_string i) }
| ['a'-'z']+ as v
{ printf "var %s\n" v;VAR v }
| '{'
{ LCURL }
| '}'
{ RCURL }
| '['
{ printf "["; LBRACK }
| ']'
{ printf "]"; RBRACK }
| ';'
{ SEMI }
| ':'
{ COLON }
| ','
{ COMMA }
| '"'
{ QUOTE }
| eof
{ EOF }
| _
{ raise (Failure (sprintf "At offset %d: unexpected character.\n"
(Lexing.lexeme_start lexbuf))) }
And a simple ml file is:
open Core.Std
open Printf
let rec read_all ic =
try
let ln = input_line ic in
ln ^ read_all ic
with End_of_file -> "";;
let () =
let linebuf = Lexing.from_string (read_all stdin) in
try
Parser.main Lexer.token linebuf;
printf "Done"
with
| Failure msg ->
fprintf stderr "%s%!" msg
| Parser.Error ->
fprintf stderr "Syntax error at offset %d.\n%!"
(Lexing.lexeme_start linebuf)
Edit: Here's a Makefile. parser.mly, lexer.mll, and interpreter.ml are the second, third, and fourth files above.
all: HB lexer.cmx parser.cmx interpreter.cmx
@true
HB: interpreter.cmx
ocamlfind ocamlopt -o HB -linkpkg -package core -package core_kernel \
-thread -w -10 parser.cmx lexer.cmx interpreter.cmx
interpreter.cmx: lexer.cmx
ocamlfind ocamlopt -package core -package core_kernel -thread -w -10 \
-c interpreter.ml
lexer.cmx: lexer.ml parser.cmx
ocamlfind ocamlopt -c lexer.ml
parser.cmx: parser.mly
menhir --ocamlc "ocamlfind ocamlc" --infer --base parser parser.mly
ocamlfind ocamlc -c parser.mli
ocamlfind ocamlopt -c parser.ml
lexer.ml: lexer.mll
ocamllex lexer.mll
clean:
@rm HB *.o *.cmi *.cmx lexer.ml parser.ml parser.mli 2>/dev/null || true
and here's making / running it, where test.in is the first one above.
$ mk;HB < test.in
ocamllex lexer.mll
menhir --ocamlc "ocamlfind ocamlc" --infer --base parser parser.mly
15 states, 286 transitions, table size 1234 bytes
Warning: 3 states have shift/reduce conflicts.
Warning: 3 shift/reduce conflicts were arbitrarily resolved.
ocamlfind ocamlc -c parser.mli
ocamlfind ocamlopt -c parser.ml
ocamlfind ocamlopt -c lexer.ml
ocamlfind ocamlopt -package core -package core_kernel -thread -w -10 \
-c interpreter.ml
ocamlfind ocamlopt -o HB -linkpkg -package core -package core_kernel \
-thread -w -10 parser.cmx lexer.cmx interpreter.cmx
Syntax error at offset 45.
var foo
var min
qvar min
var max
qvar max
var foo
[var min
]qvar min
Edit 2: I ended up just adding | e = VAR LBRACK v = q_var RBRACK
{ GetMap(v,LookupVar(e)) }
as a special case to my parser. So, problem solved?
Looking back at this, I believe the reason it doesn't work is that the grammar is not LR(1) and therefore cannot be parsed accurately by Menhir. Determining if foo["min"]
is the start of v = VAR LBRACK f = q_var RBRACK ASSIGN e = exp
or is e = exp LBRACK v = q_var RBRACK
, requires us to lookahead four or so symbols and Menhir as an LR(1) parser only looks ahead one.