yacc fails to parse simpe shell grammar

Question

I am writing a simple grammar for a shell using yacc/lex. I want my grammar to recognize pipelines, which have the following form: command1 | command2 | ... | commandn. I am able to regonize a single command, with the simple_command rule as the start non-terminal in the code below. However, when I add additional rules (simple_command_list, and pipeline) to parse a pipeline, things don't work.To test the grammar, I make yacc read input from the following string: char *input = "command1 | command2 | command3 | command4\n\0", defined in the main function. When asked to parse this string, yacc only parses the first command, prints "parse error", and stops, just like so:

command "command1"
simple command 
1B
parse error

LEX CODE:

%{ 

#include <string.h> 
#include "y.tab.h" 

%} 

%%


\n      { 
                return NEWLINE; 

        } 

[ \t]   { 
                /* Discard spaces and tabs */ 
        } 

">"     { 
                return GREAT; 
        } 

"<"     { 
                return LESS; 
        } 



“|”    { 
                return PIPE; 
        } 

“&”    { 
                return AMPERSAND; 
        } 


[a-zA-Z][a-zA-Z0-9]*  { 
                /* Assume that file names have only alpha chars */ 
                yylval.str = strdup(yytext); 
                return WORD; 
        } 



.       { 
                /* Invalid character in input */ 
                return BAD_TOKEN;
         }

%%
int yywrap(void) {
return 1;
}

YACC CODE:

%{
#include <string.h>
#include <stdio.h>
int yylex(void);

void yyerror(char *);


%}

%union
{

    char *str;
    int i;

}

%token <i> AMPERSAND GREAT LESS PIPE NEWLINE BAD_TOKEN
%token <str> WORD
%start pipeline
%expect 1


%%

cmd:
    WORD
    {
        printf("command \"%s\"\n", $1);

    }
;

arg:
    WORD
    {

         printf("argument \"%s\"\n", $1);
    }

;

arg_list:
    arg_list arg
    {

        //printf(" argument list: \n");
    }
    | // empty


;

simple_command:
    cmd arg_list
    {

         printf("simple command \n");
    }

;

simple_command_list:
         simple_command_list PIPE simple_command
        {
            printf("1A\n");
        }
        | simple_command
        {
            printf("1B\n");
        }
;

pipeline:
    simple_command_list NEWLINE
    {

    printf("p-A\n");
    }
    | NEWLINE
    {
        printf("p-B\n");
    }
    ;


%%


void yyerror(char *s) {
    fprintf(stderr, "%s\n", s);
}


int main(void) {
    // read input from a string
    //YY_BUFFER_STATE *bp;
    struct yy_buffer_state *bp;

    char *input = "command1 | command2 | command3 | command4\n\0";

    //  connect input buffer to specified string
    bp = yy_scan_string(input);

    // read from the buffer
    yy_switch_to_buffer(bp);


    // parse 
    yyparse();

    // delete the buffer
    yy_delete_buffer(bp);

    // delete the string (or not)


    return 0;
}

Answer 1

Your lex source file contains unicode characters like “ (U-201C LEFT DOUBLE QUOTATION MARK) and ” (U-201D RIGHT DOUBLE QUOTATION MARK), which lex does not recogize as quotes, so is looking for an input sequence containing that 7-byte utf-8 sequence rather than a single byte |.

Replace those with Ascii " characters and it should work.

If you enable debugging with the --debug option to bison, you should see what tokens it is getting and what rules it is shifting and reducing. In your case, getting a BAD_TOKEN for the |...