Search code examples
apache-sparkapache-spark-sqldatabrickscatalyst-optimizer

What happened to the ability to visualize query plans in a Databricks notebook?

There is an old (year 2014) talk on Youtube where the speaker visualized a query plan right inside a Databricks notebook. Here is the screenshot:

enter image description here

I am using databricks runtime 5.5 LTS ML and whenever I try to call viz on a query plan, I get this kind of error:

error: value viz is not a member of org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

Has this feature been deprecated in Spark 2.0+ or do I need to install/import other libraries to get this feature?

Solution

  • I don't think there is a visualization of it anymore, but they do explain SQL queries, which I think is what you are looking for.

    spark.sql("your SQL query").explain(true)

    or 

    yourDataframe.explain(true)