Search code examples
pythoncompressionpicklejoblib

Python - Error opening pickle file created with job lib


I've inherited a dataset stored in a pickle file, which was to the best of my knowledge, created in python 3.

I am trying to open the pickle file in python using the following code:

f = open('file.pkl', 'rb')
cl = pickle.loads(f)

When I do so, I get the following error:

TypeError: a bytes-like object is required, not '_io.BufferedReader'

A similar error is thrown when I try to read it in with only an 'r' argument.

I can see that the pickel file has been created with joblib, which I have never used before. I can't find any documentation saying if joblib pickel files need to be opened a different way.

Here is the code to create the pickle file:

joblib.dump(list_of_dataframes, 'file.pkl', compress=3)

Any help is greatly appreciated!

EDIT

This also won't open when I try to load directly from joblib:

with open(pickle_file) as f:
    data = joblib.load(f)

Which throws the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Solution

  • I found the answer, and it wasn't what I expected! (Thank you @JohnGordon for earlier help.) My file was corrupted because it was too large to do a direct pull from GitHub. To fix this, I used Git LFS and did a git lfs pull instead of a git pull. Now unpickling works without errors.