How to set multiline rule in Monaco Editor

I am looking at the sandbox for Monaco Editor here: https://microsoft.github.io/monaco-editor/playground.html#extending-language-services-custom-languages

I'm trying to support the following case:

"A single line quote"
"But also a slightly weird
multi line quote"

I need to support both because the language I'm working with support both. Now, the rule I set up looks like this:

// Register a tokens provider for the language
monaco.languages.setMonarchTokensProvider('mySpecialLanguage', {
    tokenizer: {
        root: [
            [/"(?:[^"\\]|\\.\n{0,})*"/gi, 'my-string'],
        ]
    }
});

// Define a new theme that contains only rules that match this language
monaco.editor.defineTheme('myCoolTheme', {
    base: 'vs',
    inherit: false,
    rules: [
        { token: 'my-string', foreground: '0000FF' },
    ]
});

The problem is that the multi-line quote isn't working. I'm suspecting that Monaco parses the text/code line-by-line which causes the problem, but surely I can't be alone with this issue. Is there a flag or something I have to set?

Solution

The trick is to work with tokenizer states and the tokenizer stack. Check the Monarch documentation at https://microsoft.github.io/monaco-editor/monarch.html.

In my tokenizer I have 3 rules for the 3 possible (MySQL) string types, which are single quote, double quote and backtick quote:

        stringsSql: [
            [/'/, { token: "string.quoted.single.sql", next: "@stringSingleSql" }],
            [/"/, { token: "string.quoted.double.sql", next: "@stringDoubleSql" }],
            [/`/, { token: "string.quoted.other.sql", next: "@stringBacktickSql" }],
        ],
        stringSingleSql: [
            [/[^']+/, "string.quoted.single.sql"],
            [/(''|\\')/, "string.quoted.single.sql"],
            [/'/, { token: "string.quoted.single.sql", next: "@pop" }],
        ],
        stringDoubleSql: [
            [/[^"]+/, "string.quoted.double.sql"],
            [/(''|\\")/, "string.quoted.double.sql"],
            [/"/, { token: "string.quoted.double.sql", next: "@pop" }],
        ],
        stringBacktickSql: [
            [/[^`]+/, "string.quoted.other.sql"],
            [/``/, "string.quoted.other.sql"],
            [/`/, { token: "string.quoted.other.sql", next: "@pop" }],
        ],

When a quote char is found the associated token type for it is set and I push a new state on the tokenizer stack. The tokenizer continues in this state (e.g. stringSingelSql) until it finds another quote. Regardless how many lines are processed by this, they all get the string token type assigned. Once the closing quote was found the tokenizer stack returns to the previous state.

You can also handle escaping of quote characters and double quote characters (as shown in the example).