Save a pickle file without using open(filename, 'wb')

Question

I'm trying to create an azure machine learning pipeline with tensorflow estimator step, and the output is a pickle file, but I'm not able to save the file. Any clues on how to fix this?

Starting the daemon thread to refresh tokens in background for process with pid = 138
Traceback (most recent call last):
  File "train.py", line 64, in <module>
    joblib.dump(data, output_path)
  File "/azureml-envs/azureml_cb121c6a55dfafffd79cca4bedd52636/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 504, in dump
    with open(filename, 'wb') as f:
IsADirectoryError: [Errno 21] Is a directory: '/mnt/batch/tasks/shared/LS_root/jobs/azureml/mounts/workspaceblobstore/azureml/output_data/data.pkl'

Answer 1

It looks like you're trying to save a file data.pkl to a directory /mnt/batch/tasks/shared/LS_root/jobs/azureml/mounts/workspaceblobstore/azureml/output_data that already has a directory (not file) named data.pkl. Try removing the directory data.pkl first before re-running. If the error appears again, you must be accidentally creating the directory data.pkl somewhere in your code.