Search code examples
pythonpandasdataframeconcatenation

Iterate through excel files and sheets and concatenate in Python


Say I have a folder which have multiple excel files with extension xlsx or xls, they share same header column a, b, c, d, e except some empty sheet in several files.

I want to iterate all the files and sheets (except for empty sheets) and concatenate them into one sheet of one file output.xlsx.

I have iterated through all excel files and append them to one file, but how could I iterate through all the sheets of each files if they have more than one sheets?

I need to integrate two block of code below into one. Thanks for your help.

import pandas as pd
import numpy as np
import glob

path = os.getcwd()
files = os.listdir(path)
files

df = pd.DataFrame()

# method 1

excel_files = [f for f in files if f[-4:] == 'xlsx' or f[-3:] == 'xls']
excel_files

for f in excel_files:
    data = pd.read_excel(f)
    df = df.append(data)

# method 2

for f in glob.glob("*.xlsx" or "*.xls"):
    data = pd.read_excel(f)
    df = df.append(data, ignore_index=True)

# save the data frame
writer = pd.ExcelWriter('output.xlsx')
df.to_excel(writer, 'sheet1')
writer.save()

For one file to concatenate multiple sheets:

file = pd.ExcelFile('file.xlsx')

names = file.sheet_names  # read all sheet names

df = pd.concat([file.parse(name) for name in names])

Solution

  • import pandas as pd
    
    path = os.getcwd()
    files = os.listdir(path)
    files
    
    excel_files = [file for file in files if '.xls' in file]
    excel_files
    
    def create_df_from_excel(file_name):
        file = pd.ExcelFile(file_name)
    
        names = file.sheet_names
    
        return pd.concat([file.parse(name) for name in names])
    
    df = pd.concat(
        [create_df_from_excel(xl) for xl in excel_files]
    )
    
    # save the data frame
    writer = pd.ExcelWriter('output.xlsx')
    df.to_excel(writer, 'sheet1')
    writer.save()