Spark version : 2.1
For example, in pyspark, i create a list
test_list = [['Hello', 'world'], ['I', 'am', 'fine']]
then how to create a dataframe form the test_list, where the dataframe's type is like below:
DataFrame[words: array<string>]
here is how -
from pyspark.sql.types import *
cSchema = StructType([StructField("WordList", ArrayType(StringType()))])
# notice extra square brackets around each element of list
test_list = [['Hello', 'world']], [['I', 'am', 'fine']]
df = spark.createDataFrame(test_list,schema=cSchema)