Search code examples
parsingbisonlemon

Using Lemon parser with custom token values


I am porting an old grammar to lemon and I have all the terminal symbols already defined in a header file; I would like to use them with those values instead of the ones generated in parser.h by lemon: is that possible?
Overwriting parser.h is completly useless because that's just a mirror of what happens internally, the matched values would keep being the same.
(Since lemon shares a lot of code with Bison I think that a solution for bison would solve the problem in lemon too)


Solution

  • With bison, you can manually assign values to tokens by declaring them in the %token directive (%token TOK 263, for example). However, that option is not available in lemon (as far as I know).

    In any event, the above does not really meet your request, because it doesn't allow you to read the token values from an external header file. That would not be a trivial requirement for a parser generator. In order to build the parse tables, the parser generator, whether it is bison or lemon, must actually know the value associated with each token, and the task of parsing a header to extract that information is well beyond the complexity of a parser generator; it would need an embedded C parser.

    I would recommend just letting the parser generator generate the header file, and then using it instead of the definitions in your existing header file. The only cost (afaics) is that you need to recompile any parts of the project which reference the token values, which would typically be limited to the lexer.