Search code examples
python-3.xmachine-learningdatasetgoogle-colaboratoryapp-config

Problem in running python program with Colab for training dataset with config file

I want to run a python program train.py with colab so I use following line code.

 !python /content/nameprogram/train.py --config /content/nameprogram/configs/config.yml

In config file, address of datasets and other parameters are placed and I show you part of line codes for address of dataset for train and test and other parameters in config.yml

#data
data_obj: "/content/nameprogram/data/data_{}_obj.lmdb"
data_q: "/content/nameprogram/data/data_{}_q.lmdb"

max_seq_length: 20
max_obj_num: 4
max_q_num: 25
batch_size: 24

Dataset and config file and the .py file are located in these paths and when I run above line code, config file is detected and could be seen all the parameters in that config file.

My problem is that when I run above line code, dataset didn't load for program with this address in config file and couldn't recognize dataset However, the config file is loaded correctly and loaded other parameters in that file. it gives me following error and can't load dataset for this program for training.

self.env = lmdb.open( lmdb.Error: /content/nameprogram/data/data_trainval_obj.lmdb: No such file or directory

How could I solve my problem? Is it possible that I should have initial addresses differently? (If yes how to change?)

Solution

  • There are number of .mdb files in the subsets of LMDB file. when I downloaded .lmdb files from dropbox with following code

    !wget -O /content/nameprogram/data/data_trainval_obj.lmdb https://www.dropbox.com/sh/data_trainval_obj.lmdb?dl=0

    That file saved as named.lmdb?dl=0 and when I renamed as named.lmdb and removed ?dl=0 I couldn't see that .mdb subfiles again. so, I downloaded that .mdb files one by one. And after I do that, lmdb.open("path") could recognize directory.