Search code examples
scalaapache-spark

Getting max value out of a dataframe column of timestamp in scala/spark

I am working with a spark dataframe where it contains the entire timestamp values from the Column 'IMG_CREATED_DT'.I have used collectAsList() and toString() method to get the values as List and converting in to String. But I am not getting how to fetch the max value out of it.Please guide me on this.

 val query_new =s"""(select IMG_CREATED_DT from 
 ${conf.get(UNCAppConstants.DB2_SCHEMA)}.$table)"""

 println(query_new)

 val db2_op=ConnectionUtilities_v.createDataFrame(src_props,srcConfig.url,query_new)

val t3 = db2_op.select("IMG_CREATED_DT").collectAsList().toString

How to get the max value out of t3.

Solution

  • You can calculate the max value form dataframe itself. Try the following sample. 

    val t3 = db2_op.agg(max("IMG_CREATED_DT").as("maxVal")).take(1)(0).get(0)