Search code examples
apache-sparkpysparkapache-spark-sqldatabricksaws-databricks

Set default timezone in Databricks to ESTA

I'm running code in Databricks but somehow unixtimestamp values are not different. On debugging I get to know, because databricks is configured with UTC timezone. Please advise how can I change default timezone to EST in Databricks.

Solution

  • You can set it in the cluster -> configuration -> Advanced Option -> spark, set the spark parameter:

    spark.sql.session.timeZone Asia/Hongkong

    It will impact any sql statement time zone.

    If you want to set the time zone for all data nodes, you need to add an initial script and set the unix time zone. so that some thing like the python datetime.dateime.now(), would be the set timezone.