How to flatten array of struct?

Question

How to change schema in PySpark from this

|-- id: string (nullable = true)
|-- device: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- device_vendor: string (nullable = true)
|    |    |-- device_name: string (nullable = true)
|    |    |-- device_manufacturer: string (nullable = true)

to this

|-- id: string (nullable = true)
|-- device_vendor: string (nullable = true)
|-- device_name: string (nullable = true)
|-- device_manufacturer: string (nullable = true)

Answer 1

Use a combination of explode and the * selector:

import pyspark.sql.functions as F

df_flat = df.withColumn('device_exploded', F.explode('device'))
            .select('id', 'device_exploded.*')

df_flat.printSchema()
# root
#  |-- id: string (nullable = true)
#  |-- device_vendor: string (nullable = true)
#  |-- device_name: string (nullable = true)
#  |-- device_manufacturer: string (nullable = true)

explode creates a separate record for each element of the array-valued column, repeating the value(s) of the other column(s). The column.* selector turns all fields of the struct-valued column into separate columns.