Search code examples
javascriptcompiler-errorsnewlinemoo

My programing language's Lexer will not process NEWLINE (done with moo js)


I'm creating a new programing language just for fun and to know understand languages at more basic level. I started with writing a lexer with moo js and everything is working fine except for NEWLINE. I tried many things but it just won't resolve. I even tried by copying exact piece of code from moo js's documentation but still no help.

Lexer Code:

const moo = require("moo");
const lexer = moo.compile({
whitespace: /[ \t]+/,
// comment: /\/\/.*?$/,
number:  /0|[1-9][0-9]*/,
string:  /"(?:\\["\\]|[^\n"\\])*"/,
leftParen:  '(',
rightParen:  ')',
// keyword: ['while', 'if', 'else', 'moo', 'cows'],
assignmentOp: "=",
identifier: /[a-zA-Z_][a-zA-Z0-9_]*/,
newline: { match: /\n/, lineBreaks: true },
});
module.exports = lexer;

text-lexer code:

const fs = require("fs").promises;
const lexer = require("./lexer");

async function main() {
const code = (await fs.readFile("example1.hin")).toString();
lexer.reset(code);

let token;
while (true) {
    token = lexer.next();
    if (token) {
        console.log("Got token", token);
    } else {
        break;
    }
  }
}
main().catch(err => console.log(err.stack));

test-example:

n = 4
m = 6

Solution

  • I also faced the same issue and solution was to change the regex for new line as Windows and Linux handled newline differently ( To find out more. check this out ). The one you mentioned :

    newline: { match: /\n/, lineBreaks: true },
    

    works for Linux

    To work on both, use this regex:

    newline: { match: /\r?\n/, lineBreaks: true },