Search code examples
pythonpandasexport-to-csvrows

Combine multiple CSVs, skip first 9 rows for each file

In my jupyter notebook I am usually able to combine multiple .csv format files in a shared folder into one .csv with the following code that I did not originally write.

I now have files that have 9 rows of header/material that I want to omit, but don't know how to make it work with this format. The files are pretty big individually as well.

How can I combine all the .csv files together and remove the 9 preceding rows? Thanks for the help!

import os
import glob
import pandas as pd 
import csv

extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]

#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ])

#export to csv
combined_csv.to_csv( "combined_csv.csv", index=False, encoding='utf-8-sig')
Solution

  • pandas.read_csv takes an argument skiprows.

    Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file.

    E.g.,

    combined_csv = pd.concat([pd.read_csv(f, skiprows=9) for f in all_filenames ])