How to get depth images from the camera in pyBullet

Question

In pyBullet, I have struggled a bit with generating a dataset. What I want to achieve is to get pictures of what the camera is seeing: img = p.getCameraImage(224, 224, renderer=p.ER_BULLET_HARDWARE_OPENGL)

Basically: to get the images that are seen in Synthetic Camera RGB data and Synthetic Camera Depth Data (especially this one), which are the camera windows you can see in the following picture on the left.

        p.resetDebugVisualizerCamera(cameraDistance=0.5, cameraYaw=yaw, cameraPitch=pitch, cameraTargetPosition=[center_x, center_y, 0.785])
        img = p.getCameraImage(224, 224, renderer=p.ER_BULLET_HARDWARE_OPENGL)
        rgbBuffer = img[2]
        depthBuffer = img[3]
        list_of_rgbs.append(rgbBuffer)
        list_of_depths.append(depthBuffer)

        rgbim = Image.fromarray(rgbBuffer)
        depim = Image.fromarray(depthBuffer)

        rgbim.save('test_img/rgbtest'+str(counter)+'.jpg')
        depim.save('test_img/depth'+str(counter)+'.tiff')

       
        counter += 1

I already run the following, so I don't know if it is related to the settings. p.configureDebugVisualizer(p.COV_ENABLE_DEPTH_BUFFER_PREVIEW, 1)
I have tried several methods because the depth part is complicated. I don't understand if it needs to be treated separately because of the pixel color information or if I need to work with the project matrixes and view matrixes. I need to save it as a .tiff because I get some cannot save F to png errors. I tried playing a bit with the bit information but acomplished nothing. In case you asked,

# depthBuffer[depthBuffer > 65535] = 65535
# im_uint16 = np.round(depthBuffer).astype(np.uint16)
# depthBuffer = im_uint16 

The following is an example of the the .tiff image

And to end, just to remark that these depth images keep changing (looking at all of them, then to the RGB and passing again to the depth images, shows different images regardless of being the same image. I have never ever seen something like this before.

Answer 1

I thought "I managed to fix this some time ago, might as well post the answer found".

The data structure of img has to be taken into account!

    img = p.getCameraImage(224, 224, shadow = False, renderer=p.ER_BULLET_HARDWARE_OPENGL)
    rgb_opengl = (np.reshape(img[2], (IMG_SIZE, IMG_SIZE, 4)))
    depth_buffer_opengl = np.reshape(img[3], [IMG_SIZE, IMG_SIZE])
    depth_opengl = far * near / (far - (far - near) * depth_buffer_opengl)
    seg_opengl = np.reshape(img[4], [IMG_SIZE, IMG_SIZE]) * 1. / 255.

    rgbim = Image.fromarray(rgb_opengl)
    rgbim_no_alpha = rgbim.convert('RGB')

    rgbim_no_alpha.save('dataset/'+obj_name+'/'+ obj_name +'_rgb_'+str(counter)+'.jpg')
    # plt.imshow(depth_buffer_opengl)
    plt.imsave('dataset/'+obj_name+'/'+ obj_name+'_depth_'+str(counter)+'.jpg', depth_buffer_opengl)
    # plt.show()

Final Images: