Extract matrix from batch, represented as Tensor

Question

I'm using Tensorflow in C++. I'm using a trained model to extract patches from an input image.

My output tensor (after session Run) is outputs and has in outputs[0] a batch of N patches, MxMxD.

auto patches = outputs[0].tensor<float, 4>();

Now, I want to display this images using OpenCV, in particular, I want to use the cv::eigen2cv function, that given an Eigen::Matrix gives me a cv::Mat.

The problem is that I need to loop over this output tensor and for each element, extract an Eigen::Matrix.

I've tried the proposed solution, here: https://stackoverflow.com/a/39475756/2891324 but I can't even compile the code because of Eigen error: EIGEN_STATIC_ASSERT_VECTOR_ONLY:

const auto patch_side = 256;
int batch_id = 0;
using Matrix =
    Eigen::Matrix<float, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor>;
for (auto i = 0; i < grid_shape.second; ++i) {
    for (auto j = 0; j < grid_shape.first; ++j) {
        auto size = patch_side * patch_side * depth;
        auto map =
            Eigen::Map<Matrix>(patches.data() + batch_id * size, size);
        // Eigen::Matrix4f m = map;
        Matrix m = map;

        cv::Mat p;
        cv::eigen2cv(m, p);
        cv::imshow("p", p);
        cv::waitKey();
        /*
        ml(cv::Rect(i * patch_side, j * patch_side, patch_side,
                    patch_side)) =
            cv::Mat(cv::Size(patch_side, patch_side), CV_32F, patches.slice
                    patches.data() + (i + j) * patch_side * patch_side);
                    */

        std::cout << "i,j" << i << "," << j << "\n";
        batch_id++;
    }
}

So, how can I get a Matrix from a Eigen::Tensor (or tf::Tensor) that I'll be able to use into cv::eigen2cv?

Answer 1

Looking a bit around the documenation of OpenCV, and considering TensorFlow tensors are always row-major, you should be able to just do something like:

const auto patch_side = 256;
int batch_id = 0;
for (auto i = 0; i < grid_shape.second; ++i) {
    for (auto j = 0; j < grid_shape.first; ++j) {
        auto size = patch_side * patch_side * depth;
        cv::Mat p(patch_side, patch_side, CV_32FC(depth), patches.data() + batch_id * size);
        cv::imshow("p", p);
        cv::waitKey();
        /*
        ml(cv::Rect(i * patch_side, j * patch_side, patch_side,
                    patch_side)) =
            cv::Mat(cv::Size(patch_side, patch_side), CV_32F, patches.slice
                    patches.data() + (i + j) * patch_side * patch_side);
                    */

        std::cout << "i,j" << i << "," << j << "\n";
        batch_id++;
    }
}

Note this does not copy the tensor data but rather creates a cv::Mat that points to it.