I am looking for a state machine representation for transformation of a JavaScript string into tokens, Is there any other alternative representations? any state machine or automata's representation?
JavaScript can't be tokenized with a finite-state automaton, because the goal symbol for each successive token is determined by the state of the syntactic parse at that point.