Transform list in a dataframe (same row, different columns) in Pyspark

Question

I got one list from a dataframe's column:

list_recs = [row[0] for row in df_recs.select("name").collect()]

The list looks like this:

Out[243]: ['COL-4560', 'D65-2242', 'D18-4751', 'D68-3303']

I want to transform it in a new dataframe, which value in one different column. I tried doing this:

from pyspark.sql import Row
rdd = sc.parallelize(list_recs)
recs = rdd.map(lambda x: Row(SKU=str(x[0]), REC_01=str(x[1]), REC_02=str(x[2]), REC_03=str(x[3])))#, REC_04=str(x[4]), REC_0=str(x[5])))
schemaRecs = sqlContext.createDataFrame(recs)

But the outcome I'm getting is:

+---+------+------+------+
|SKU|REC_01|REC_02|REC_03|
+---+------+------+------+
|  C|     O|     L|     -|
|  D|     6|     5|     -|
|  D|     1|     8|     -|
|  D|     6|     8|     -|
+---+------+------+------+

What I wanted:

+----------+-------------+-------------+-------------+
|SKU       |REC_01       |REC_02       |REC_03       |
+----------+-------------+-------------+-------------+
|  COL-4560|     D65-2242|     D18-4751|     D68-3303|
+----------+-------------+-------------+-------------+

I've also tried spark.createDataFrame(lista_recs, StringType()) but got all the items in the same column.

Thank you in advance.

Answer 1

Define schema and use spark.createDataFrame()

list_recs=['COL-4560', 'D65-2242', 'D18-4751', 'D68-3303']

from pyspark.sql.functions import *
from pyspark.sql.types import *

schema = StructType([StructField("SKU", StringType(), True), StructField("REC_01", StringType(), True), StructField("REC_02", StringType(), True), StructField("REC_03", StringType(), True)])

spark.createDataFrame([list_recs],schema).show()
#+--------+--------+--------+--------+
#|     SKU|  REC_01|  REC_02|  REC_03|
#+--------+--------+--------+--------+
#|COL-4560|D65-2242|D18-4751|D68-3303|
#+--------+--------+--------+--------+