Search code examples
scalaapache-sparktimestampdatetime-formatdatetime-conversion

Spark convert TimestampType to String of format yyyyMMddHHmm

In my dataframe I have a column of TimestampType format of '2019-03-16T16:54:42.968Z' I would like to convert this to a StringType column with a format of '201903161654' instead of a generic timestamp column. This is because I need to partition several directories based on the string formatted timestamp, if I partition on the timestamp column it creates special characters when creating the directory.

Is there any api I can use in spark to convert the Timestamp column to a string type with the format above?

Solution

  • Use the date_format function: date_format(date/timestamp/string ts, string fmt).

    Converts a date/timestamp/string to a value of string in the format specified by the date format fmt. Supported formats are Java SimpleDateFormat formats. The second argument fmt should be constant. Example: date_format('2015-04-08', 'y') is "2015".