Spark 2.0 final was released last week - when can we expect to see it on Dataproc? Spark 2.0 uses Scala 2.11 as default - does this mean that Dataproc will use Scala 2.11?
Scala 2.11 is actually the feature we are most eagerly awaiting. Scala 2.10 is getting old and we have a couple of libraries we would like to use that we cannot because they don't support 2.10.
To follow this up, Cloud Dataproc 1.1, newly released this week, has Spark 2.0 and Scala 2.11 and is intended for production uses. See the release notes for more details!