I am a beginner in machine learning, I am trying to do my own object detection using my own dataset. However, it would be more practical if the object is labeled with polygon shaped bound. yet tensorflow object detection API can only accept bounding box.
So is it possible to modify the API such that, it can accept polygon labeled dataset??
Yes, it is possible. You have to give directory of the training set. But bounding box is recommended because during inference time, you will get bounding box around the object detected. You can see an example here in tensorflow.org.
For labeling you can use LabelImg, which is very simple and easy to use also will increase the detection accuracy.