I want to know the function performed by tokenize in codemirror.
codemirror highlights text by calling a tokenizer function, passing it a context ("state"), and a pointer to the current location in the file that needs to be highlighted ("stream"). The job of this function is to advance the stream past the next token, and to return the type of the token. This is described fairly well in the codemirror api documentation here: http://codemirror.net/doc/manual.html#modeapi
In the case of xml.js (which you referenced in a comment), it has multiple tokenizer functions. Depending on the context, it will set the "tokenize" attribute of the state to refer to one of the tokenizer functions. Then it will use whichever function is pointed by by state.tokenize to find the next token in the stream.