In my project I am forced to use these packages:
com.sparkjava:spark-core:2.3, which ends up using jetty-server:9.3.2.v20150730org.apache.spark:spark-core_2.10:1.2.0, which ends up using jetty-server:8.1.14.v20131031Note that com.sparkjava and org.apache.spark have nothing to do with each other. They are called both spark funnily.
The issue here is that both jetty versions are incompatible, so if I force jetty 8.X the system crashes, if I force jetty 9.X the system crashes again, I get java.lang.NoClassDefFoundError: org/eclipse/jetty/server/ServerConnector in one case and java.lang.NoClassDefFoundError: org/eclipse/jetty/server/bio/SocketConnector in the other.
What I am expected to do in such a situation ?
Note: I've tried to shadow jetty, but the dependency manager resolves just one (9.X by default, or 8.X if I force it) and then it shadows it, so it's really not helping.
It would be exceedingly difficult to resolve this situation.
Jetty 8.1 is about 4 major version behind Jetty 9.3, which represents many hundreds of releases of difference.
Note: Jetty versioning is [servlet_support].[major_ver].[minor_ver].
Jetty 8.x is Servlet 3.0, while Jetty 9.x is Servlet 3.1
The architecture of the connectors has evolved tremendously in that time frame, from being old school blocking Sockets in Jetty 8 to no blocking connectors at all in Jetty 9, with Jetty 9 needing to evolve the connectors to support features in TLS/1.2, and ALPN in order to properly support HTTP/2, and the internal I/O handling to support the new Servlet 3.1 Async I/O feature set.
Solution #1:
You won't be able to have both versions running in the same VM without some sort of classloader isolation, and careful configuration to ensure they don't claim the same resources (listening ports, temp files, etc)
Solution #2:
Upgrade (or downgrade) one or the other spark dependency till you hit a common jetty version. (Spark_2.11 / 2.0.0 seems to support Jetty 9.2.x)
Solution #3:
Apache Spark is open source, go submit a patch that upgrades its use of Jetty to 9.3 (this might be difficult as Apache Spark isn't ready to use Java 8 yet, which is a requirement for Jetty 9.3)