Search code examples
rstringdatesparklyr

Convert String to timestamp in Sparklyr

I'd like to convert String to time stamp, I used the below code but it added 4 hours to the timestamp, I really appreciate if you could help me to figure it out the issue

I used Sparklyr.

fds<- fds %>%  
  mutate(time2 = from_unixtime(
    unix_timestamp(transaction_time, "yyyy-MM-ddHH:mm:ss"), "yyyy-MM-dd HH:mm:ss"))
fds<- fds %>% mutate(transaction_time_2 = to_timestamp(time2))

transaction_time is string with this format '2018-05-3016:00:40' by using from_unixtime, it has been converted to ''2018-05-30 16:00:40' and by using to_timestamp it has been converted to 2018-05-30 20:00:40. everything is correct except 20 hour, I still expected 16.

Solution

  • I encountered the same problem, I think it relates to the timezone settings try the following at the top of your program, then the problem should disappear:

    sparklyr::invoke_static(sc, "java.util.TimeZone",  "getTimeZone", "GMT") %>%
  sparklyr::invoke_static(sc, "java.util.TimeZone", "setDefault", .)

    check out this answer for more details :

    changing the JVM timezone in sparklyr