machine-learning ubuntu-14.04 deep-learning caffe lmdb

How to generate coorectlly lmdb files from own dataset using caffe ?

I'm trying to to generate lmdbs to train my CNN using Caffe.

I've used as data set Stanford Dogs Dataset.After downloading the data set , I prepared the training data set et test data set.I have saved the configuration into the files train.txt and val.txt.

Then , I tried to generate the lmdb files by those two following commands on my console :

GLOG_logtostderr=1 /home/bme/caffe/build/tools/convert_imageset \
    --resize_height=200 --resize_width=200 --shuffle  \
    /home/bme/caffe/data/dogs/data/ \
    /home/bme/caffe/data/dogs/dogs_data/train.txt \
    /home/bme/caffe/examples/dogs/dogs_train_lmdb

GLOG_logtostderr=1 /home/bme/caffe/build/tools/convert_imageset \
    --resize_height=200 --resize_width=200 --shuffle  \
    /home/bme/caffe/data/dogs/data/ \
    /home/bme/caffe/data/dogs/dogs_data/val.txt \
    /home/bme/caffe/examples/dogs/dogs_val_lmdb

But , it produced those errors for the first commands :

I0331 18:13:39.625823  8810 convert_imageset.cpp:86] Shuffling data
I0331 18:13:39.627516  8810 convert_imageset.cpp:89] A total of 17493 images.
I0331 18:13:39.627708  8810 db_lmdb.cpp:35] Opened lmdb /home/bme/caffe/examples/dogs/dogs_train_lmdb
E0331 18:13:39.627734  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/108_n02111889-Samoyed17.jpg 108   
E0331 18:13:39.627743  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/12_n02104365-schipperke145.jpg 12   
E0331 18:13:39.627753  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/17_n02113712-miniature_poodle107.jpg 17   
E0331 18:13:39.627760  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/12_n02104365-schipperke63.jpg 12   
E0331 18:13:39.627768  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/97_n02105056-groenendael102.jpg 97   
E0331 18:13:39.627774  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/74_n02096585-Boston_bull38.jpg 74   
E0331 18:13:39.627781  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/4_n02113799-standard_poodle145.jpg 4   
E0331 18:13:39.627789  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/22_n02097047-miniature_schnauzer27.jpg 22   
E0331 18:13:39.627797  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/80_n02107312-miniature_pinscher167.jpg 80   
E0331 18:13:39.627805  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/95_n02091134-whippet42.jpg 95

The errors for the second command are:

I0331 18:13:39.625823  8810 convert_imageset.cpp:86] Shuffling data
I0331 18:13:39.627516  8810 convert_imageset.cpp:89] A total of 17493 images.
I0331 18:13:39.627708  8810 db_lmdb.cpp:35] Opened lmdb /home/bme/caffe/examples/dogs/dogs_train_lmdb
E0331 18:13:39.627734  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/108_n02111889-Samoyed17.jpg 108   
E0331 18:13:39.627743  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/12_n02104365-schipperke145.jpg 12   
E0331 18:13:39.627753  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/17_n02113712-miniature_poodle107.jpg 17   
E0331 18:13:39.627760  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/12_n02104365-schipperke63.jpg 12   
E0331 18:13:39.627768  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/97_n02105056-groenendael102.jpg 97   
E0331 18:13:39.627774  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/74_n02096585-Boston_bull38.jpg 74   
E0331 18:13:39.627781  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/4_n02113799-standard_poodle145.jpg 4   
E0331 18:13:39.627789  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/22_n02097047-miniature_schnauzer27.jpg 22   
E0331 18:13:39.627797  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/80_n02107312-miniature_pinscher167.jpg 80   
E0331 18:13:39.627805  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/95_n02091134-whippet42.jpg 95   
E0331 18:13:39.627812  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/56_n02101556-clumber86.jpg 56   
E0331 18:13:39.627820  8810 io.cpp:80] Could not open or find file /home/bme/caffe/data/dogs/data/26_n02085782-Japanese_spaniel36.jpg 26

I tried to solve the issue by changing the permissions of the files train.txt and val.txt . But the same errors remained .So , any suggestions to fix this issue.

Thanks

Solution

The outputs you posted are both for the training set generation (they are the same). It seems like out of 17,493 images you have 12 corrupted files that caffe cannot find/open.
Check that these files exists, and that they are not corrupted.