Pytorch is not working with trained model+pretrained model (Intel Open Vino)

Question

def PeopleBox(PeopleNet,frame):
frameHeight=frame.shape[0]
frameWidth=frame.shape[1]
blob=cv2.dnn.blobFromImage(frame, 1.0, (672,384), swapRB=False, crop=True)
PeopleNet.setInput(blob)
detection=PeopleNet.forward()
bboxs=[]
for i in range(detection.shape[2]):
    confidence=detection[0,0,i,2]
    if confidence>0.7:
        x1=int(detection[0,0,i,3]*frameWidth)
        y1=int(detection[0,0,i,4]*frameHeight)
        x2=int(detection[0,0,i,5]*frameWidth)
        y2=int(detection[0,0,i,6]*frameHeight)
        bboxs.append([x1,y1,x2,y2])
        cv2.rectangle(frame, (x1,y1),(x2,y2),(0,255,0), 8)
return frame, bboxs



PeopleBin = (r"C:\Users\directory\T1\pedestrian-detection-adas-0002.bin")
PeopleXml = (r"C:\Users\directory\T1\pedestrian-detection-adas-0002.xml")
HelmetBin = (r"C:\Users\dc\StructVGGV1_output\VGG16_V1_40epohc_LR0_00008_batch4_A96_V77.bin")
HelmetXml = (r"C:\Users\dc\StructVGGV1_output\VGG16_V1_40epohc_LR0_00008_batch4_A96_V77.xml")
PeopleNet=cv2.dnn.readNet(PeopleXml, PeopleBin)
HelmetNet=cv2.dnn.readNet(HelmetXml,HelmetBin)
List = ['NoPersonHoldingHelmet', 'PersonHoldingHelmet']
video=cv2.VideoCapture(0)

while True:
ret,frame=video.read()
framee,bboxs=PeopleBox(PeopleNet,frame)
for bbox in bboxs:
    ########check
    blob=cv2.dnn.blobFromImage(framee, 1.0, (224,224), swapRB=False, crop = True) 
    HelmetNet.setInput(blob)
    ###########
    HelmetPred=HelmetNet.forward()
    Helmet=List[HelmetPred[0].argmax()]
    #label="{},{}".format(Helmet,HelmetPred)
    label="{}".format(Helmet)
    if label=="NoPersonHoldingHelmet":
        cv2.rectangle(framee, (bbox[0], bbox[1]), (bbox[2],bbox[3]), (255,0,255), 8)
    if label == "PersonHoldingHelmet":
        cv2.rectangle(framee, (bbox[0], bbox[1]), (bbox[2],bbox[3]), (255,255,0), 8)
        
    cv2.putText(framee, label, (bbox[0]-70, bbox[1]-10), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (255,255,255), 2,cv2.LINE_AA)

cv2.imshow("Helmet_Vs_NoHelmet",framee)
cv2.imshow("Helmet_Vs_NoHelmet",framee)
k=cv2.waitKey(1)
if k==ord('q'):
    break
video.release()
cv2.destroyAllWindows()

I trained my model with Pytorch using google colab. After I trained my model, I converted my model into IR using Intel Open Vino (https://docs.openvino.ai/2022.3/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_PyTorch.html) After that, I combined pretrained pedestrian model (https://docs.openvino.ai/2021.4/omz_models_model_pedestrian_detection_adas_0002.html) and my model together (object detection + recognition like license plate recognition/detection). However, the combined model is not working correctly... I checked my model's accuracy and it was near 96 percent. Therefore, I am not sure what is causing the issue. I have been hard stuck in this for so long time.

I uploaded my colab below I used to train my model enter link description here

Answer 1

I tested and compared your model with OpenVINO pretrained model pedestrian-detection-adas-0002 since you mentioned that this is the reference to your DL model development.

I inferred both model with Object Detection Python Demo ( I get this from OMZ repo)

My finding is that your model does not have the correct wrapper as the OV model.The pedestrian-detection-adas-0002 uses network based on SSD framework with tuned MobileNet v1 as a feature extractor.

Perhaps this is the part that you need to cater to your custom model.