Search code examples
pythonpandaslistcsvcumsum

Find cumulative sum of a list while reading data from csv file

I want to find cumulative sum of the list

I am reading 30 similar csv files located in the folder deg_pvsyst_runs. Files are named as:

 "Energy_Base_Year00_-0.6%modqual.csv",
 "Energy_Base_Year01_-0.3%modqual.csv",  ......,  
 "Energy_Base_Year30_-8.4%modqual.csv

"

I am reading E_Grid Column (After skipping rows 0-9, 11 and 12 which has header info) in the each of the above csv files. Please suggest

a) If there is a shorter/easier way to read csv while skipping rows as opposed to what I have done (skiprows=[0,1,2,3,4,5,6,7,8,9,11,12])

b) If using itertools method or cumsum function would be a better option to calculate the cumulative sum of list called Cumulative_Annual_Deg

CODE:

import os, csv, re
import pandas as pd
from itertools import accumulate

Year, Degradation, Mean_EP, Annual_Deg, Cumulative_Annual_Deg =[],[],[],[],[]


cwd = os.getcwd()
csv_files = [f for f in os.listdir(cwd + '\\' + 'deg_pvsyst_runs') if f.endswith('.csv')]

for i,j in enumerate(csv_files):
    df = pd.read_csv(os.getcwd() + "\\deg_pvsyst_runs\\" + j, skiprows=[0,1,2,3,4,5,6,7,8,9,11,12])
    Mean_EP.append(df['E_Grid'].sum()/10**6)
    Annual_Deg.append((Mean_EP[i-1] - Mean_EP[i])/Mean_EP[i-1])
    Cumulative_Annual_Deg.append(list(accumulate(Annual_Deg[i])))

ERROR:

Cumulative_Annual_Deg.append(list(accumulate(Annual_Deg[i])))

   TypeError: 'numpy.float64' object is not iterable

EDIT:

Annual_Deg is list consisting of delta (% change) calculate between the csv files. I want to calculate Cumulative_Annual_Deg to calculate cumulative sum.

Desired output :

  • Annual_Mean_EP = [9559.88, 9533.31, 9506.67,...,8731.85]

    Annual_Deg = [0, 0.00278, 0.00279,...,0.00297]

    Cumulative_Annual_Deg' = [0, 0.00278, 0.00557, ..., 0.08661]

OR Screenshot

enter image description here

Solution

  • The cumulative sum could be calculated after the list is calculated. So the accumulate line should be brought outside the for loop. Replace following line

        Cumulative_Annual_Deg.append(list(accumulate(Annual_Deg[i])))

    with 

    Cumulative_Annual_Deg = list(accumulate(Annual_Deg))

    You can find more info about itertools on https://docs.python.org/3/library/itertools.html#itertool-functions