select from a column made of string array pyspark or python high order function multiple values

Question

i have a table like this and i want to create a new column based on what is listed on the column example

df.withcolumn('good',.when('java' or 'php' isin ['booksIntereste']).lit(1).otherwise(0))

desired output when it contain java or php get 1 else 0

Answer 1

You can directly use a Higher Order Function - array_contains for this , additionally you can browse through this article to understand more

Data Preparation

d = {
    'name':['James','Washington','Robert','Micheal'],
    'booksInterested':[['Java','C#','Python'],[],['PHP','Java'],['Java']]
}

sparkDF = sql.createDataFrame(pd.DataFrame(d))

sparkDF.show()

+----------+------------------+
|      name|   booksInterested|
+----------+------------------+
|     James|[Java, C#, Python]|
|Washington|                []|
|    Robert|       [PHP, Java]|
|   Micheal|            [Java]|
+----------+------------------+

Array Contains

sparkDF = sparkDF.withColumn('good',F.array_contains(F.col('booksInterested'), 'Java'))

+----------+------------------+-----+
|      name|   booksInterested| good|
+----------+------------------+-----+
|     James|[Java, C#, Python]| true|
|Washington|                []|false|
|    Robert|       [PHP, Java]| true|
|   Micheal|            [Java]| true|
+----------+------------------+-----+

ForAll Array Contains - Multiple

sparkDF = sparkDF.withColumn('good_multiple',F.forall(F.col('booksInterested'), lambda x: x.isin(['Java','Python','PHP'])))

sparkDF.show()

+----------+------------------+-----+-------------+
|      name|   booksInterested| good|good_multiple|
+----------+------------------+-----+-------------+
|     James|[Java, C#, Python]| true|        false|
|Washington|                []|false|         true|
|    Robert|       [PHP, Java]| true|         true|
|   Micheal|            [Java]| true|         true|
+----------+------------------+-----+-------------+