Blank screen when generating point cloud from image with Open3D
So I tried creating a point cloud with the Open3D library in python and in the end, it's basically just the 2 lines as referenced in here, but when I run my code (see below) all I get is a white screen popping up. I've ran it in Jupyter notebook, but running it in a python script from console didn't change anything nor it threw an error. I should mention that I created the images in Blender and saved it as OpenExr, meaning that the depth value range between 0 and 4 (I've truncated it to 4 for the background). You can see that they are proper images below and I could also transform them to Open3D pictures and display them without problems.
Edit (27-03-2020): Added complete minimal example
import OpenEXR
import numpy as np
import array
import matplotlib.pyplot as plt
import open3d as o3d
%matplotlib inline
exr_img = OpenEXR.InputFile('frame0100.exr')
depth_img = array.array('f', exr_img.channel('View Layer.Depth.Z'))
r_img = array.array('f', exr_img.channel('View Layer.Combined.R'))
g_img = array.array('f', exr_img.channel('View Layer.Combined.G'))
b_img = array.array('f', exr_img.channel('View Layer.Combined.B'))
def reshape_img(img):
return np.array(img).reshape(720, 1280)
img_array = np.dstack((reshape_img(r_img),
reshape_img(g_img),
reshape_img(b_img),
reshape_img(depth_img)))
#Background returns very large value, truncate it to 4
img_array[img_array == 10000000000.0] = 4
colour = o3d.geometry.Image(np.array(img_array[:, :, :3]))
depth = o3d.geometry.Image(np.array(img_array[:, :, 3]*1000).astype('uint16'))
o3d.draw_geometries([depth])
pinhole_cam = o3d.open3d.camera.PinholeCameraIntrinsic(width= 1280, height=720, cx=640,cy=360,fx=500,fy=500)
rgbd = o3d.create_rgbd_image_from_color_and_depth(colour, depth, convert_rgb_to_intensity = False, depth_scale=1000)
pcd = o3d.create_point_cloud_from_rgbd_image(rgbd, pinhole_cam)
o3d.draw_geometries([pcd])
Please overlook the hacky way of importing the data, as I am new to Open3D and produced the data myself, I did it step-by-step for checks and for confirming the data integrity
I assume it might have to do with my parameters for the pinhole camera. Tbh, I have no idea what would be the proper parameters except that cy, cy should be the centre of the image and fx, fy should be sensible. As my depth values are in Blender metres but Open3D apparently expects millimetres, the scaling should make sense.
I'd appreciate it if you could give me any help debugging it. But if you were to point me in the direction of a better working library to create 3D point clouds from images I wouldn't mind either. The documentation I found with Open3D is lacking at best.
In short, Open3D expects your 3-channel color image to be of uint8 type.
Otherwise, it would return an empty point cloud, resulting in the blank window you see.
Update 2020-3-27, late night in my time zone:)
Now that you have provided your code, let's dive in!
From your function names, I guess you are using Open3D 0.7.0 or something like that. The code I provided is in 0.9.0. Some function names changed and new functionality added in.
When I run your code in 0.9.0 (after some minor modifications of course), there's a RuntimeError:
RuntimeError: [Open3D ERROR] [CreatePointCloudFromRGBDImage] Unsupported image format.
And we can see from the Open3D 0.9.0 source that your color image must be of 3 channels and take only 1 byte each (uint8) or be of 1 channel and take 4 bytes (float, that means intensity image):
std::shared_ptr<PointCloud> PointCloud::CreateFromRGBDImage(
const RGBDImage &image,
const camera::PinholeCameraIntrinsic &intrinsic,
const Eigen::Matrix4d &extrinsic /* = Eigen::Matrix4d::Identity()*/,
bool project_valid_depth_only) {
if (image.depth_.num_of_channels_ == 1 &&
image.depth_.bytes_per_channel_ == 4) {
if (image.color_.bytes_per_channel_ == 1 &&
image.color_.num_of_channels_ == 3) {
return CreatePointCloudFromRGBDImageT<uint8_t, 3>(
image, intrinsic, extrinsic, project_valid_depth_only);
} else if (image.color_.bytes_per_channel_ == 4 &&
image.color_.num_of_channels_ == 1) {
return CreatePointCloudFromRGBDImageT<float, 1>(
image, intrinsic, extrinsic, project_valid_depth_only);
}
}
utility::LogError(
"[CreatePointCloudFromRGBDImage] Unsupported image format.");
return std::make_shared<PointCloud>();
}
Otherwise, there'll be errors like I encountered.
However, in the version of 0.7.0, the source code is:
std::shared_ptr<PointCloud> CreatePointCloudFromRGBDImage(
const RGBDImage &image,
const camera::PinholeCameraIntrinsic &intrinsic,
const Eigen::Matrix4d &extrinsic /* = Eigen::Matrix4d::Identity()*/) {
if (image.depth_.num_of_channels_ == 1 &&
image.depth_.bytes_per_channel_ == 4) {
if (image.color_.bytes_per_channel_ == 1 &&
image.color_.num_of_channels_ == 3) {
return CreatePointCloudFromRGBDImageT<uint8_t, 3>(image, intrinsic,
extrinsic);
} else if (image.color_.bytes_per_channel_ == 4 &&
image.color_.num_of_channels_ == 1) {
return CreatePointCloudFromRGBDImageT<float, 1>(image, intrinsic,
extrinsic);
}
}
utility::PrintDebug(
"[CreatePointCloudFromRGBDImage] Unsupported image format.\n");
return std::make_shared<PointCloud>();
}
That means Open3D still does not support it, but it would only warn you. And only in debug mode!
After that, it will return an empty point cloud. (Actually both versions do this.) That explains the blank window.
Now you should know, you can make convert_rgb_to_intensity=True and succeed. Though you still should normalize your color image first.
Or you can convert the color image to be in range [0, 255] and of type uint8.
Both will work.
Now I hope all is clear. Hooray!
P.S. Actually I usually found Open3D source code to be easy to read. And if you know C++, you could read it whenever something like this happens.
Update 2020-3-27:
I cannot reproduce your result and I don't know why it happened (you should provide a minimal reproducible example).
Using the image you provided in the comment, I wrote the following code and it works well. If it still doesn't work on your computer, maybe your Open3D is broken.
(I'm not familiar with .exr images, hence the data extraction might be ugly :)
import Imath
import array
import OpenEXR
import numpy as np
import open3d as o3d
# extract data from exr files
f = OpenEXR.InputFile('frame.exr')
FLOAT = Imath.PixelType(Imath.PixelType.FLOAT)
cs = list(f.header()['channels'].keys()) # channels
data = [np.array(array.array('f', f.channel(c, FLOAT))) for c in cs]
data = [d.reshape(720, 1280) for d in data]
rgb = np.concatenate([data[i][:, :, np.newaxis] for i in [3, 2, 1]], axis=-1)
# rgb /= np.max(rgb) # this will result in a much darker image
np.clip(rgb, 0, 1.0) # to better visualize as HDR is not supported?
# get rgbd image
img = o3d.geometry.Image((rgb * 255).astype(np.uint8))
depth = o3d.geometry.Image((data[-1] * 1000).astype(np.uint16))
rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(img, depth, convert_rgb_to_intensity=False)
# some guessed intrinsics
intr = o3d.open3d.camera.PinholeCameraIntrinsic(1280, 720, fx=570, fy=570, cx=640, cy=360)
# get point cloud and visualize
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd, intr)
o3d.visualization.draw_geometries([pcd])
And the result is:
Original answer:
You have misunderstood the meaning of depth_scale.
Use this line:
depth = o3d.geometry.Image(np.array(img_array[:, :, 3]*1000).astype('uint16'))
The Open3D documentation said the depth values will first be scaled and then truncated. Actually it means the pixel values in your depth image will first divide this number rather than multiply, as you can see in the Open3D source:
std::shared_ptr<Image> Image::ConvertDepthToFloatImage(
double depth_scale /* = 1000.0*/, double depth_trunc /* = 3.0*/) const {
// don't need warning message about image type
// as we call CreateFloatImage
auto output = CreateFloatImage();
for (int y = 0; y < output->height_; y++) {
for (int x = 0; x < output->width_; x++) {
float *p = output->PointerAt<float>(x, y);
*p /= (float)depth_scale;
if (*p >= depth_trunc) *p = 0.0f;
}
}
return output;
}
Actually we usually take it for granted that values in depth images should be integers (I guess that's why Open3D did not point that out clearly in their documentation), since floating-point images are less common.
You cannot store 1.34 meters in .png images, since they will lose precision. As a result, we store 1340 in depth images and later processes will first convert it back to 1.34.
As for camera intrinsics for your depth image, I guess there'll be configuration parameters in Blender when you create it. I'm not familiar with Blender, so I'll not talk about it. However, if you do not understand general camera intrinsics, you might want to take a look at this.