Search code examples
pythonchunks

split data into chunks

Maybe this question is simple, but I am trying to find a way how to do it in an automatic way, suppose I have a data frame and I want to split it into chunks and save them using names based on chunk. I managed how to split, but how I can save it in .cvs using names data_1, data_2 and so on ... PS. I just need chunks, so I am not using sklearn methods here.

import numpy as np
import pandas as pd
df = pd.DataFrame({
    'x_values':np.random.randn(400),
    'y_values':np.random.randn(400),
})

sample = 100
N = int(len(df)/sample)
frames = [df.iloc[i*sample:(i+1)*sample] for i in range(N+1) ]
Solution

  • You can use the to_csv built-in method:

    [df.iloc[i*sample:(i+1)*sample].to_csv('data_'+str(i)+'.csv') for i in range(N+1) ]

    This will create individual dataframes named data_0.csv, data_1.csv, etc...