Search code examples
memory-leaksnvidiaonnxonnxruntime

Memory leak (CPU's RAM) when using onnxruntime on GPU


I'm using the Insightface library from Pypi (https://pypi.org/project/insightface/), the source code is here: https://github.com/deepinsight/insightface/blob/master/python-package/insightface/model_zoo/scrfd.py.

When I run it on my GPU there is a severe memory leak of the CPU's RAM, over 40 GB until I stopped it (not the GPU memory).

here is my script:

import insightface
import cv2
import time

model = insightface.app.FaceAnalysis()

# It happens only when using GPU !!!
ctx_id = 0

image_path = "my-face-image.jpg"
image = cv2.imread(image_path)

model.prepare(ctx_id = ctx_id, det_thresh=0.3, det_size=[416, 416])

detector =  model.models["detection"]

for i in range(100000):
    start_t = time.time()
    bboxes, landmarks = detector.detect(image)
    end_t = time.time()
    print('Detection time: {}'.format(end_t - start_t))

print('DONE')

My setup is (inside docker):

  • Docker Base Image - nvidia/cuda:11.0.3-cudnn8-devel-ubuntu18.04
  • Nvidia Driver - 465.27
  • python - 3.6.9
  • insightface==0.3.8
  • mxnet==1.8.0.post0
  • mxnet-cu110==2.0.0a0
  • numpy==1.18.5
  • onnx==1.9.0
  • onnxruntime-gpu==1.8.1

Solution

  • I managed to solve it with the following setup:

    • Ubuntu-20.04
    • Python-3.8
    • Nvidia-470
    • Cuda-11.3
    • Cudnn-8
    • mxnet==1.8.0.post0
    • onnx==1.9.0
    • onnxruntime-gpu==1.8.1
    • insightface==0.4