Search code examples
scalahadoopapache-sparkhiveparquet

Spark: what options can be passed with DataFrame.saveAsTable or DataFrameWriter.options?


Neither the developer nor the API documentation includes any reference about what options can be passed in DataFrame.saveAsTable or DataFrameWriter.options and they would affect the saving of a Hive table.

My hope is that in the answers to this question we can aggregate information that would be helpful to Spark developers who want more control over how Spark saves tables and, perhaps, provide a foundation for improving Spark's documentation.


Solution

  • The reason you don't see options documented anywhere is that they are format-specific and developers can keep creating custom write formats with a new set of options.

    However, for few supported formats I have listed the options as mentioned in the spark code itself: