Search code examples
pythonpandasapache-spark-sqlpyspark-pandas

Function to take a list of spark dataframe and convert to pandas then csv


import pyspark
            
dfs=[df1,df2,df3,df4,df5,df6,df7,df8,df9,df10,df1,df12,df13,df14,df15]
    
for x in dfs:
    y=x.toPandas()
    y.to_csv("D:/data")

This is what I wrote, but I actually want the function to take this list and convert every df into a pandas df and then convert it to csv and save it in the order as it appears on dfs list and save it to a particular directory in the order of name. Is there a possible way to write such function? PS D:/data is just an imaginary path and is used for explanation.


Solution

  • If you will convert a dataframe to a csv, you still need to state it in df.to_csv. So, try:

    for x in dfs:
           y=x.toPandas()
           y.to_csv(f"D:/data/df{dfs.index(x) + 1}.csv")
    

    I set it as df{dfs.index(x) + 1} so that the file names will be df1, df2, ... etc.