What does the merge parameter of a token do?

Question

In the ACE overview on creating syntax highlighters, under the section 'Defining States', the token definitions include the parameter merge: true. But I cannot find an explanation of what this does. What is its purpose?

Excerpt from example:

this.$rules = {
    "start" : [ {
        token : "text",
        merge : true,
        regex : "<\\!\\[CDATA\\[",
        next : "cdata"
    },
    "cdata" : [ {
        token : "text",
        regex : "\\]\\]>",
        next : "start"
    }, {
        token : "text",
        merge : true,
        regex : "\\s+"
    }, {
        token : "text",
        merge : true,
        regex : ".+"
    } ]
}; 

Answer 1

Figured out by trial and error: Setting the merge property to true in a token will cause the token to be merged with the following token, both in the token list in memory and as a rendered span in the UI DOM, but only if the following token also evaluates to the same token type. I'm using this to merge the compound SQL tokens IS NULL and IS NOT NULL:

In the following rules, the 3 tokens [is + (spaces) + null], or the 5 tokens [is + (spaces) + not + (spaces) + null] will be merged into a single token. If is is followed by something other that not or null, that something will be flagged as invalid. It will not be merged because, even though the previous token as still in merge mode, the resulting token class (invalid) for the next token is different.

this.$rules = {
    "start": [{
        token : "keyword.operator",
        regex: "[iI][sS]\\b",
        merge: true,
        next: "is_keyword"
    }],
    "is_keyword": [
        {
            token: "keyword.operator",
            regex: "\\s+",
            merge: true
        }, {
            token: "keyword.operator",
            regex: "[nN][oO][tT]\\b",
            merge: true
        }, {
            token: "keyword.operator",
            regex: "[nN][uU][lL][lL]\\b",
            next: "start"
        }, {
            token: "invalid",
            regex: ".+",
            next: "start"
        }
    ]
};