Search code examples
javabashmavenapache-sparkjava-17

Does a way exist through jar, manifest... to setup --add-exports, --add-opens directives definitively, to avoid passing them on command line?


I encounter a problem that the whole Spark community has:
when I switch from Java 11 to 17, Spark (for me the 3.3.2 version) fails with:

java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0xe25b2fe) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0xe25b2fe

The solution is to :

  • Change your Maven launch of unit tests,
  • Change your Dockerfiles,
  • Change your java command lines in any script involving directly or indirectly your classes or Spark,
  • And ask all your customers to do the same

to go from:

java -jar target/myProgram.jar

to :

java --add-exports java.base/sun.nio.ch=ALL-UNNAMED \
   --add-opens java.base/java.util=ALL-UNNAMED \
   --add-opens java.base/java.nio=ALL-UNNAMED \
   --add-opens java.base/java.lang=ALL-UNNAMED \
   --add-opens java.base/java.lang.invoke=ALL-UNNAMED \
   -jar myProgram.jar

And you understand that the response is, at 100%:
"No, let's stay on Java 11.
Spark (or Scala) doesn't handle Java 17 correctly, making it unusable with it
"

Does an alternative solution exist, that would allow me to, like I would do savagely a:
System.setProperty("my.property", "myValue")
if I was really willing so,

to setup at the earliest place of my build process,
something that would ensure that for/from any launch of the classes, jars I'm producing here - whenever they are intended for tests, runtime, containerization... -,

the underlying java command launching them will assume the
--add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.invoke=ALL-UNNAMED
to come along with it, avoiding for anyone the clumsy (and sometimes impossible task) of correcting all the java command lines everywhere?

Because it's a definitive stumbling block for the migration from Java 11 to Java 17.


Solution

  • JEP 261: Module System has this to say:

    Two new JDK-specific [emphasis added] JAR-file manifest attributes are defined to correspond to the --add-exports and --add-opens command-line options:

    • Add-Exports: <module>/<package>( <module>/<package>)*
    • Add-Opens: <module>/<package>( <module>/<package>)*

    The value of each attribute is a space-separated list of slash-separated module-name/package-name pairs. A <module>/<package> pair in the value of an Add-Exports attribute has the same meaning as the command-line option --add-exports <module>/<package>=ALL-UNNAMED. A <module>/<package> pair in the value of an Add-Opens attribute has the same meaning as the command-line option --add-opens <module>/<package>=ALL-UNNAMED.

    Each attribute can occur at most once, in the main section of a MANIFEST.MF file. A particular pair can be listed more than once. If a specified module was not resolved, or if a specified package does not exist, then the corresponding pair is ignored. These attributes are interpreted only in the main executable JAR file of an application, i.e., in the JAR file specified to the -jar option of the Java run-time launcher; they are ignored in all other JAR files. [emphasis added]

    As emphasized, these attributes are JDK-specific. They do not appear to be part of the JAR file specification. And they only work with the JAR file specified with the -jar option.