The entrypoint to my issues was an exception:
[info] Cause: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.4
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:64)
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:51)
[info] at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
[info] at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:745)
[info] at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
[info] at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
[info] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
[info] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
It occurred after upgrade of Spark. I managed to overcome this problem, however I still don't understand root cause. It looks like problem with Spark dependencies, please look:
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/2.4.7
It needs com.fasterxml.jackson.core » jackson-databind 2.6.7.3.
No problem.
Further, it needs as well:
https://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.12/3.5.3
No problem.
However, https://mvnrepository.com/artifact/org.json4s/json4s-jackson_2.12/3.5.3 requires https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind/2.8.4
Now, we can see that spark-2.4.7 introduced dependencies conflict between jackson-databind itself. Where am I wrong in my reasoning?
As you can see in the parent-pom, com.fasterxml.jackson.core.* is excluded from json4s-jackson:
https://github.com/apache/spark/blob/v2.4.7/pom.xml#L755
So, there is no conflict after all