How to predict a mini-batch examples in C++ of MXNet?

Question

In python interface,we can use a mini-batch examples to make prediction like net([[1,2],[3,4],[5,6]]).

But in C++,I can't find a way to do this.

Suppose calling the net to predict a single example needs 10ms. If there is 10000 examples needs to make prediction, that is 100s

void OneInputOneOutputPredict(PredictorHandle pred_hnd, std::vector<mx_float> vector_data, std::vector<mx_float> &output)
{
    MXPredSetInput(pred_hnd, "data", vector_data.data(), vector_data.size());

    // Do Predict Forward
    MXPredForward(pred_hnd);

    mx_uint output_index = 0;
    mx_uint *shape = 0;
    mx_uint shape_len;
    MXPredGetOutputShape(pred_hnd, output_index, &shape, &shape_len);
    size_t size = 1;
    for (mx_uint i = 0; i < shape_len; ++i) size *= shape[i];

    std::vector<float> data(size);
    assert(0 == MXPredGetOutput(pred_hnd, output_index, &(data[0]), size));
    output = data;
}

//very long time
for(int step=0;step<10000;step++)
    OneInputOneOutputPredict(pred_hnd, vector_data, vector_label);

Could we use vectorize the code or something way in C++ that make it fast in prediction?

Answer 1

originally input_shape_data looks like this

const mx_uint input_shape_data[4] = {1, static_cast<mx_uint>(data_len)};

now if I want to predict a mini-batch(batch-size 3)

const mx_uint input_shape_data[4] = {3, static_cast<mx_uint>(data_len)};

If using seq2seq model.If data looks like [[1,2],[3,4],[5,6]],now only flatten it to a list {1,2,3,4,5,6} , then everything is OK