How to properly resolve "cannot find symbol" after "Run JFlex Generator" in Intellij Language Plugin Tutorial?

Question

I'm going through Custom Language Support Tutorial and receiving compilation error on the last section of part 4.

The error is: cannot find symbol zzAtBOL = true; location: class SimpleLexer

If I comment this line, the next error is: cannot find symbol if (!zzEOFDone) { location: class SimpleLexer

Commenting out both references to zzAtBOL and zzEOFDone "resolves" the issue, and I'm able to runIde. But obviously it's not the solution to the problem.

So, the question is how to make "Run JFlex Generator" generate valid sources in my case?

---

Some details:

Using Intellij Idea Build #IU-223.8836.41, built on March 10, 2023

Using Kotlin instead of Java for plugin implementation.

com.example.sampleplugin.SimpleLanguage.kt:

package com.example.sampleplugin

// imports

object SimpleLanguage : Language("Simple")

object SimpleFileType : LanguageFileType(SimpleLanguage) {
    override fun getName() = "Simple File"

    override fun getDescription() = "Simple language file"

    override fun getDefaultExtension() = "simple"

    override fun getIcon() =
        IconLoader.getIcon("/icons/jar-gray.png", SimpleLanguage::class.java)
}

object SimpleLexerAdapter : FlexAdapter(SimpleLexer(null))

class SimpleFile(viewProvider: FileViewProvider) : PsiFileBase(viewProvider, SimpleLanguage) {
    override fun getFileType() = SimpleFileType
}

object SimpleTokenSets {
    val identifier = TokenSet.create(SimpleTypes.KEY)
    val comments = TokenSet.create(SimpleTypes.COMMENT)
}

object SimpleParserDefinition : ParserDefinition {
    override fun createLexer(project: Project?) = SimpleLexerAdapter

    override fun createParser(project: Project?) = SimpleParser()

    override fun getFileNodeType() = IFileElementType(SimpleLanguage)

    override fun getCommentTokens() = SimpleTokenSets.comments

    override fun getStringLiteralElements(): TokenSet = TokenSet.EMPTY

    override fun createElement(node: ASTNode?): PsiElement = SimpleTypes.Factory.createElement(node)

    override fun createFile(viewProvider: FileViewProvider) = SimpleFile(viewProvider)
}

com/example/sampleplugin/psi.SimplePsi.kt:

package com.example.sampleplugin.psi

// imports

class SimpleTokenType(debugName: String) : IElementType(debugName, SimpleLanguage)

class SimpleElementType(debugName: String) : IElementType(debugName, SimpleLanguage)

.bnf and .flex files I've copied from the tutorial as is (only changed packages):

{
  parserClass="com.example.sampleplugin.parser.SimpleParser"

  extends="com.intellij.extapi.psi.ASTWrapperPsiElement"

  psiClassPrefix="Simple"
  psiImplClassSuffix="Impl"
  psiPackage="com.example.sampleplugin.psi"
  psiImplPackage="com.example.sampleplugin.psi.impl"

  elementTypeHolderClass="com.example.sampleplugin.psi.SimpleTypes"
  elementTypeClass="com.example.sampleplugin.psi.SimpleElementType"
  tokenTypeClass="com.example.sampleplugin.psi.SimpleTokenType"
}

simpleFile ::= item_*

private item_ ::= (property|COMMENT|CRLF)

property ::= (KEY? SEPARATOR VALUE?) | KEY

package com.example.sampleplugin;

import com.intellij.lexer.FlexLexer;
import com.intellij.psi.tree.IElementType;
import com.example.sampleplugin.psi.SimpleTypes;
import com.intellij.psi.TokenType;

%%

%class SimpleLexer
%implements FlexLexer
%unicode
%function advance
%type IElementType
%eof{  return;
%eof}

CRLF=\R
WHITE_SPACE=[\ \n\t\f]
FIRST_VALUE_CHARACTER=[^ \n\f\\] | "\\"{CRLF} | "\\".
VALUE_CHARACTER=[^\n\f\\] | "\\"{CRLF} | "\\".
END_OF_LINE_COMMENT=("#"|"!")[^\r\n]*
SEPARATOR=[:=]
KEY_CHARACTER=[^:=\ \n\t\f\\] | "\\ "

%state WAITING_VALUE

%%

<YYINITIAL> {END_OF_LINE_COMMENT}                           { yybegin(YYINITIAL); return SimpleTypes.COMMENT; }

<YYINITIAL> {KEY_CHARACTER}+                                { yybegin(YYINITIAL); return SimpleTypes.KEY; }

<YYINITIAL> {SEPARATOR}                                     { yybegin(WAITING_VALUE); return SimpleTypes.SEPARATOR; }

<WAITING_VALUE> {CRLF}({CRLF}|{WHITE_SPACE})+               { yybegin(YYINITIAL); return TokenType.WHITE_SPACE; }

<WAITING_VALUE> {WHITE_SPACE}+                              { yybegin(WAITING_VALUE); return TokenType.WHITE_SPACE; }

<WAITING_VALUE> {FIRST_VALUE_CHARACTER}{VALUE_CHARACTER}*   { yybegin(YYINITIAL); return SimpleTypes.VALUE; }

({CRLF}|{WHITE_SPACE})+                                     { yybegin(YYINITIAL); return TokenType.WHITE_SPACE; }

[^]                                                         { return TokenType.BAD_CHARACTER; }

Answer 1

The problem is in the version divergence between JFlex and idea-flex.skeleton used by the Grammar-Kit Intellij plugin. It seems that the plugin at the moment takes the older jflex-1.7.0-2.jar with a newer idea-flex.skeleton.

And in this newer idea-flex.skeleton zzAtBOL and zzEOFDone are removed (see IC commit).

So, the solution for me is to download and use newer JFlex from the CLI:

java -Xmx512m -Dfile.encoding=UTF-8 \
  -jar jflex-1.9.1.jar -skel idea-flex.skeleton \
  -d src/main/gen/com/example/sampleplugin \
  src/main/kotlin/com/example/sampleplugin/Simple.flex