How do I condense a pandas data frame where the rows are the months and I'm trying to condense them into years?

Question

So I have a dataframe

https://docs.google.com/spreadsheets/d/19ssG8bvkZKVDR6V5yU9fZVRJbJNfTTEYmWqLwmDwBa0/edit#gid=0

This is the out put that my code gives.

Here is the code:

from yahoofinancials import YahooFinancials
import pandas as pd
import datetime as datetime

df = pd.read_excel('C:/Users/User/Downloads/Div Tickers.xlsx', sheet_name='Sheet1')

tickers_list = df['Ticker'].tolist()
data = pd.DataFrame(columns=tickers_list)


yahoo_financials_ecommerce = YahooFinancials(data)

ecommerce_income_statement_data = yahoo_financials_ecommerce.get_financial_stmts('annual', 'income')

data = ecommerce_income_statement_data['incomeStatementHistory']

df_dict = dict()

for ticker in tickers_list:

    df_dict[ticker] = pd.concat([pd.DataFrame(data[ticker][x]) for x in range(len(data[ticker]))],
               sort=False, join='outer', axis=1)

df = pd.concat(df_dict, sort=True)

df_l = pd.DataFrame(df.stack())
df_l.reset_index(inplace=True)
df_l.columns = ['ticker', 'financials', 'date', 'value']

df_w = df_l.pivot_table(index=['date.year', 'financials'], columns='ticker', values='value')


export_excel = df_w.to_excel(r'C:/Users/User/Downloads/Income Statement Histories.xlsx', sheet_name="Sheet1", index= True)

How would I go about condensing the months into years so that the data is comparable Year-over-Year?

Answer 1

IIUC, you need to melt, then use groupby on your date column to group by year.

#df['date'] = pd.to_datetime(df['date'])

df = pd.melt(df,id_vars=['date','financials'],var_name='ticker')

df.groupby([df['date'].dt.year,df['financials'],df['ticker']])['value'].sum().unstack()

ticker                                      AEM          AGI           ALB  \
date financials                                                              
2016 costOfRevenue                 1.030000e+09  309000000.0  1.710000e+09   
     discontinuedOperations        0.000000e+00          0.0  2.020000e+08   
     ebit                          3.360000e+08   21300000.0  5.370000e+08   
     grossProfit                   1.110000e+09  173000000.0  9.700000e+08   
     incomeBeforeTax               2.680000e+08   -7600000.0  5.750000e+08   
...                                         ...          ...           ...   
2019 researchDevelopment           0.000000e+00          0.0  5.828700e+07   
     sellingGeneralAdministrative  1.210000e+08   19800000.0  4.390000e+08   
     totalOperatingExpenses        1.650000e+09  557000000.0  2.830000e+09   
     totalOtherIncomeExpenseNet   -1.000000e+08    2900000.0 -6.900000e+07   
     totalRevenue                  2.490000e+09  683000000.0  3.590000e+09