pyspark convert millisecond timestamp to timestamp

Question

I am new in spark and i would like to retrieve timestamps in my DF.

checkpoint actual values

1594976390070

and i want :

checkpoint values without ms

1594976390070 / 1000

Actually i am using this piece of code to cast as timestamp:

# Casting dates as Timestamp
for d in dateFields:
    df= df.withColumn(d,checkpoint.cast(TimestampType()))

I wonder how to convert it into a simple timestamp.

Answer 1

Divide your column by 1000 and use F.from_unixtime to convert to timestamp type:

import pyspark.sql.functions as F

for d in dateFields:
    df = df.withColumn(d,
        (checkpoint / F.lit(1000.)).cast('timestamp')
    )