What is the best way to read large datasets with less latency for torch model?

Question

I am familiar to use some data specific layers or tools by the habits of my in use libraries like mxnet (recordIO), caffe(LMDB). They also handle data augmentation tricks as well. I decided to give a shot for Torch to see what is good or bad. From the initial gaze, there is no such data loading standard for Torch or I am missing it. Could you point some good ways to load large scale datasets for my model training?

Answer 1

Some pointers:

• twitter/torch-dataset
• fb.resnet.torch/dataloader
• Element-Research/dataload

More resources can also be found in the torch7/wiki.