Search code examples
arraysapache-sparkpysparkapache-spark-sqlnull

Removing NULL items from PySpark arrays


How to remove the null items from array(1, 2, null, 3, null)? Using the array_remove function doesn't help when we want to remove null items.


Solution

  • I used the following trick, using array_except() function:

    SELECT array_except(array(1, 2, null, 3, null), array(null)) returns [1,2,3]