Search code examples
pysparktypesdecimal-point

Pyspark Code for seeing if all floats have .0 at the end

I’m curious if there is a Pyspark Code for seeing if all floats in a column are .0 at the end or if any are another decimal number eg .1 or .2 or .5

Eg I want to be able to tell from the fake code below if all code ends in .0 or if there are any that end in different numbers.

Index Score  
  0    2.5  
  1    1.7  
  2    1.0  
  3    3.0
Solution

  • You can use the like operator. like('%.0') here checks if the float column has values that end with '.0'

    from pyspark.sql.types import *

df.withColumn("pattern_check", when(col("Score").like('%.0'), True)\
    .otherwise(False)).show()
+-----+-----+-------------+
|Index|Score|pattern_check|
+-----+-----+-------------+
|    0|  2.5|        false|
|    1|  1.7|        false|
|    2|  1.0|         true|
|    3|  3.0|         true|
+-----+-----+-------------+