Search code examples
scalaapache-sparkapache-spark-sqlhdfsunix-timestamp

Apache Spark How to convert a datetime from Australia/Melbourne time to UTC?

How do I convert a datetime string like 21/10/2021 15:15:28 in Australia/Melbourne time to UTC in Apache Spark in Scala?

Solution

  • Try this (of course you can ignore the creation of the data, I added it for you to understand the flow: columns' names and types):

    Seq("21/10/2021 15:15:28").toDF("timeStr")
      .withColumn("australiaMelbourneTime", to_timestamp(col("timeStr"), "dd/MM/yyyy HH:mm:ss"))
      .withColumn("utcTime", to_utc_timestamp(col("australiaMelbourneTime"), "Australia/Melbourne"))

    Output (tested):

    +-------------------+----------------------+-------------------+
|            timeStr|australiaMelbourneTime|            utcTime|
+-------------------+----------------------+-------------------+
|21/10/2021 15:15:28|   2021-10-21 15:15:28|2021-10-21 04:15:28|
+-------------------+----------------------+-------------------+