Search code examples
azure-synapse

Azure Apache Synapse Spark Pool Display Function Issue

When I run the display function with Databricks I get the option to download the results as CSV. I am tring to do the same with Azure Synapse Spark Pools, however I getting the error:

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-11-2d01e14> in <module>
      1 df = spark.sql("SELECT * FROM `*********")
----> 2 display(df)

~/cluster-env/env/lib/python3.8/site-packages/notebookutils/visualization/display.py in display(data, summary)
    197         log4jLogger \
    198             .error(f"display failed with error, language: python, error: {err}")
--> 199         raise err
    200 
    201     log4jLogger \

This should very simple.

enter image description here

Any thoughts?

Solution
    • I have tried using display on the dataframe where the data is being read from lake database and was able to get the desired requirement.

    enter image description here

    • Create another spark pool and try to perform the display() operation again. If this does not work, as an alternative, write the data to your datal lake storage (associated with synapse workspace) and download it from there.
    df.write.option("header",True).csv("abfss://<container>@<storage>.dfs.core.windows.net/output")

    enter image description here