Search code examples
javascripttext-parsingautomata

Is there any other way of lexing JavaScript into token except context free grammar provided in ECMAScript specification?


I am looking for a state machine representation for transformation of a JavaScript string into tokens, Is there any other alternative representations? any state machine or automata's representation?


Solution

  • JavaScript can't be tokenized with a finite-state automaton, because the goal symbol for each successive token is determined by the state of the syntactic parse at that point.