Search code examples
javaapache-sparkapache-spark-sql

Java spark Map is empty

There is a column in java spark dataset which is of type map<string,string>, how can I check with java spark if the column with this map is empty or has some value.

I want to filter dataset where column map_col has the empty values, and map_col is of type Java Map. dataset1=dataset2.filter(col("map_col").isEmpty())

Solution

  • You can use the size() function from org.apache.spark.sql.functions to check the size of the Map<String, String> and filter it according to what you need

    for example :

    DataSet<Row> myDataSet = /* Your data */;
DataSet<Row> filteredData = myDataSet.filter(size(col("map_col")).equalTo(0));

    Spark Docs Ref : https://sparkbyexamples.com/spark/spark-get-size-length-of-array-map-column/