PySpark: Concatenate Two Columns with Datatype of 'Struc' --> Error: Cannot Resolve Due to Datatype Mismatch

Question

I have a data table in PySpark that contains two columns with data type of 'struc'.

Please see sample data frame below:

word_verb                   word_noun
{_1=cook, _2=VB}            {_1=chicken, _2=NN}
{_1=pack, _2=VBN}           {_1=lunch, _2=NN}
{_1=reconnected, _2=VBN}    {_1=wifi, _2=NN}

I want to concatenate the two columns together so I can do a frequency count of the concatenated verb and noun chunk.

I tried the code below:

df = df.withColumn('word_chunk_final', F.concat(F.col('word_verb'), F.col('word_noun')))  

But I get the following error:

AnalysisException: u"cannot resolve 'concat(`word_verb`, `word_noun`)' due to data type mismatch: input to function concat should have been string, binary or array, but it's [struct<_1:string,_2:string>, struct<_1:string,_2:string>]

My desired output table is as follows. The concatenated new field would have datatype of string:

word_verb                   word_noun               word_chunk_final
{_1=cook, _2=VB}            {_1=chicken, _2=NN}     cook chicken
{_1=pack, _2=VBN}           {_1=lunch, _2=NN}       pack lunch
{_1=reconnected, _2=VBN}    {_1=wifi, _2=NN}        reconnected wifi 

Answer 1

Your code is almost there.

Assuming your schema is as follows:

df.printSchema()
#root
# |-- word_verb: struct (nullable = true)
# |    |-- _1: string (nullable = true)
# |    |-- _2: string (nullable = true)
# |-- word_noun: struct (nullable = true)
# |    |-- _1: string (nullable = true)
# |    |-- _2: string (nullable = true)

You just need to access the value of the _1 field for each column:

import pyspark.sql.functions as F

df.withColumn(
    "word_chunk_final", 
    F.concat_ws(' ', F.col('word_verb')['_1'], F.col('word_noun')['_1'])
).show()
#+-----------------+------------+----------------+
#|        word_verb|   word_noun|word_chunk_final|
#+-----------------+------------+----------------+
#|        [cook,VB]|[chicken,NN]|    cook chicken|
#|       [pack,VBN]|  [lunch,NN]|      pack lunch|
#|[reconnected,VBN]|   [wifi,NN]|reconnected wifi|
#+-----------------+------------+----------------+

Also, you should use concat_ws ("concatenate with separator") instead of concat to add the strings together with a space in between them. It's similar to how str.join works in python.