I am trying to select 500 columns from a Pyspark DatFrame. Getting error as "SyntaxError: more than 255 arguments"
Df2 = Df\
.select("col1","col2","col3",...............,"col500")
Tried below approach also, bit did not work.
cols = ["col1","col2","col3",...............,"col500"]
Df2 = Df\
.select(cols)
Both the approach is working for less than 255 columns.
Note : My Python version is 3.6
Please Advice me. Thanks.
After having conversation with @pissall, below are two workable solutions to select more than 255 columns:
Case 1:
cols = ["col1","col2","col3",...............,"col500"]
df2 = df.select(cols)
Case 2:
df.createOrReplaceTempView("df");
spark.sql("SELECT col1, col2, ..., col500 FROM df")