Adding new points to point cloud in real time - Open3D

Question

I am using Open3D to visualize point clouds in Python. Essentially, what I want to do is add another point to the point cloud programmatically and then render it in real time.

This is what I have so far. I could not find any solution to this.

In the code below, I show one possible solution, but it is not effective. The points get added and a new window is opened as soon as the first one is closed. This is not what I want. I want it to dynamically show new points, without closing and opening again. As well as the fact that a new variable is created, which I think can be problematic when working with larger and larger data sets

import open3d as o3d
import numpy as np

#Create two random points
randomPoints = np.random.rand(2, 3)

pointSet = o3d.geometry.PointCloud()

pointSet.points = o3d.utility.Vector3dVector(randomPoints)

#Visualize the two random points
o3d.visualization.draw_geometries([pointSet])

#Here I want to add more points to the pointSet
#This solution does not work effective

#Create another random set
p1 = np.random.rand(3, 3)

p2 = np.concatenate((pointSet.points, p1), axis=0)

pointSet2 = o3d.geometry.PointCloud()

pointSet2.points = o3d.utility.Vector3dVector(p2)

o3d.visualization.draw_geometries([pointSet2])

Is there any possible solution to this?

If not, what other libraries can I look at that has the ability to render new incoming points in real time.

Answer 1

New points can be added and visualized interactively to a PointCloud by extending PointCloud.points with the new coordinates.

This is because when we use numpy arrays, we need to create a Vector3dVector isntance which has the convenient method extend implemented. From the docs:

extend(*args, **kwargs)

Overloaded function.

1. extend(self: open3d.cpu.pybind.utility.Vector3dVector, L: open3d.cpu.pybind.utility.Vector3dVector) -> None

Extend the list by appending all the items in the given list

2. extend(self: open3d.cpu.pybind.utility.Vector3dVector, L: Iterable) -> None

Extend the list by appending all the items in the given list

So we can use different object instances e.g. ndarrays, Vector3dVector, lists etc.

A toy example and its result:

import open3d as o3d
import numpy as np
import time


# create visualizer and window.
vis = o3d.visualization.Visualizer()
vis.create_window(height=480, width=640)

# initialize pointcloud instance.
pcd = o3d.geometry.PointCloud()
# *optionally* add initial points
points = np.random.rand(10, 3)
pcd.points = o3d.utility.Vector3dVector(points)

# include it in the visualizer before non-blocking visualization.
vis.add_geometry(pcd)

# to add new points each dt secs.
dt = 0.01
# number of points that will be added
n_new = 10

previous_t = time.time()

# run non-blocking visualization. 
# To exit, press 'q' or click the 'x' of the window.
keep_running = True
while keep_running:
    
    if time.time() - previous_t > dt:
        # Options (uncomment each to try them out):
        # 1) extend with ndarrays.
        pcd.points.extend(np.random.rand(n_new, 3))
        
        # 2) extend with Vector3dVector instances.
        # pcd.points.extend(
        #     o3d.utility.Vector3dVector(np.random.rand(n_new, 3)))
        
        # 3) other iterables, e.g
        # pcd.points.extend(np.random.rand(n_new, 3).tolist())
        
        vis.update_geometry(pcd)
        previous_t = time.time()

    keep_running = vis.poll_events()
    vis.update_renderer()

vis.destroy_window()

---

Why not create an updated geometry and remove the old one?

For completeness, other (which I believe to be not better) alternative approach could consist on the following steps:

1. Remove the current PointCloud
2. concatenate the new points as in the OP's question
3. Create new Pointcloud and add it to the visualizer.

This yields worse perfomance and barely allows interaction with the visualization.

To see this, let's have a look to the following comparison, with the same settings (code below). Both versions run the same time (~10 secs).

Using extend	Removing and creating PointCloud
✔️ Allows interaction	❌ Difficult interaction
Mean execution time (for adding points): 0.590 ms	Mean execution time (for adding points): 1.550 ms

Code to reproduce:

import open3d as o3d
import numpy as np
import time

# Global settings.
dt = 3e-2 # to add new points each dt secs.
t_total = 10 # total time to run this script.
n_new = 10 # number of points that will be added each iteration.

#---
# 1st, using extend. Run non-blocking visualization.

# create visualizer and window.
vis = o3d.visualization.Visualizer()
vis.create_window(height=480, width=640)

# initialize pointcloud instance.
pcd = o3d.geometry.PointCloud()
# *optionally* add initial points
points = np.random.rand(10, 3)
pcd.points = o3d.utility.Vector3dVector(points)
# include it in the visualizer before non-blocking visualization.
vis.add_geometry(pcd)

exec_times = []

current_t = time.time()
t0 = current_t

while current_t - t0 < t_total:

    previous_t = time.time()

    while current_t - previous_t < dt:
        s = time.time()

        # Options (uncomment each to try it out):
        # 1) extend with ndarrays.
        pcd.points.extend(np.random.rand(n_new, 3))

        # 2) extend with Vector3dVector instances.
        # pcd.points.extend(
        #     o3d.utility.Vector3dVector(np.random.rand(n_new, 3)))

        # 3) other iterables, e.g
        # pcd.points.extend(np.random.rand(n_new, 3).tolist())

        vis.update_geometry(pcd)

        current_t = time.time()
        exec_times.append(time.time() - s)

    vis.poll_events()
    vis.update_renderer()

print(f"Using extend\t\t\t# Points: {len(pcd.points)},\n"
      f"\t\t\t\t\t\tMean execution time:{np.mean(exec_times):.5f}")

vis.destroy_window()

# ---
# 2nd, using remove + create + add PointCloud. Run non-blocking visualization.

# create visualizer and window.
vis = o3d.visualization.Visualizer()
vis.create_window(height=480, width=640)

# initialize pointcloud instance.
pcd = o3d.geometry.PointCloud()
points = np.random.rand(10, 3)
pcd.points = o3d.utility.Vector3dVector(points)
vis.add_geometry(pcd)

exec_times = []

current_t = time.time()
t0 = current_t
previous_t = current_t

while current_t - t0 < t_total:

    previous_t = time.time()

    while current_t - previous_t < dt:
        s = time.time()

        # remove, create and add new geometry.
        vis.remove_geometry(pcd)
        pcd = o3d.geometry.PointCloud()
        points = np.concatenate((points, np.random.rand(n_new, 3)))
        pcd.points = o3d.utility.Vector3dVector(points)
        vis.add_geometry(pcd)

        current_t = time.time()
        exec_times.append(time.time() - s)

    current_t = time.time()

    vis.poll_events()
    vis.update_renderer()

print(f"Without using extend\t# Points: {len(pcd.points)},\n"
      f"\t\t\t\t\t\tMean execution time:{np.mean(exec_times):.5f}")

vis.destroy_window()