Search code examples
pandasparquet

How to load Parquet file with Pandas with specified dtypes?

How can I specify the data types I want with pd.read_parquet() like I can with pd.read_csv()? When I try the same method I get an error saying pd.read_parquet has no dtype option.

dtypes={
'elapsed_time':np.int32,
'event_name':'category',
'name':'category',
'level':np.uint8,
'room_coor_x':np.float
'level_group':'category'}

dataset_df = pd.read_csv(path, dtype=dtypes)
Solution

  • It doesn't make sense to specify the dtypes for a parquet file. Unlike CSV files, parquet files store meta data with the type of each column. So the user doesn't have to specify them.

    If you want to change the type of the column you can always cast it using astype

    pd.read_parquet("my_file.parquet").astype(dtypes)