Search code examples
apache-sparkpyspark

Timestamp parsing in pyspark

df1:

Timestamp:

1995-08-01T00:00:01.000+0000

Is there a way to separate the day of the month in the timestamp column of the data frame using pyspark. Not able to provide the code, I am new to spark. I do not have a clue on how to proceed.

Solution

  • You can parse this timestamp using unix_timestamp:

    from pyspark.sql import functions as F

format = "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
df2 = df1.withColumn('Timestamp2', F.unix_timestamp('Timestamp', format).cast('timestamp'))

    Then, you can use dayofmonth in the new Timestamp column:

    df2.select(F.dayofmonth('Timestamp2'))

    More detials about these functions can be found in the pyspark functions documentation.