I need to implement a custom Mask-RCNN in C++ to perform instance segmentation on a custom dataset. Since I'm a beginner, I just know the theory, but I really don't know how to apply it.
Could you give me some guidelines to start my project? Thank you.
For a beginner, doing machine learning in C++ will be a very high bar.
Pretty much all the packages out there use python for the API. Tensorflow allows running the session API in C++, but you need to build the graph in python. And dealing with the build of tensorflow will be a pain.
Get the Mask-RCNN from its github, run it in python, understand it. Check that the license fits your need. Then, assuming your project is in C++, brush up on bindings between C++ and python. Have your C++ make calls to a python layer that imports Mask-RCNN.
Any other approach will offer significant hurdles to a beginner.