Create model with 50 million spheres

Question

I'm attempting to create a 3d model from a list of xyz coordinate data. I've created a Python script to create a small sphere and then copy it for each data point. Unfortunately, the program takes an unreasonably long time. I've calculated that it would take up to 8 days to finish creating the model. I even tried running the script with Google Compute Engine, but it still would take way too long. Is there anyway to create a model this large in a reasonable time frame? I'm open to trying other programs besides Blender. How do animation companies create huge models? There has to be some way, right?

Here is the Python script for creating the model:

import bpy
import os 

radius = 0.0005
scale = 0.001

bpy.ops.object.select_all(action='DESELECT')
bpy.ops.mesh.primitive_uv_sphere_add(size=radius)
sphere = bpy.context.object

filename = 'points.xyz'
directory = 'C:\\Users\\User1\\Desktop'
fullpath = os.path.join(os.getcwd(), filename)

print("Creating OBJ file")
sphereCount = 1

def makeSphere(x,y,z):
    global sphereCount
    print("Creating Sphere #" + str(sphereCount))
    ob = sphere.copy()
    ob.location.x = x
    ob.location.y = y
    ob.location.z = z

    bpy.context.scene.objects.link(ob)
    sphereCount += 1

with open(fullpath,"r") as f:
    for line in f:
        values = line.split()
        x = values[0]
        y = values[1]
        z = values[2]
        makeSphere(scale*float(x),scale*float(y),scale*float(z))

bpy.context.scene.update()

#Export as OBJ file

filename = 'finalObject.obj'
directory = '/home/user1'
fullpath = os.path.join(os.getcwd(), filename)

print("Exporting OBJ");

bpy.ops.export_scene.obj(filepath=fullpath)

I found this question online and tried this approach, but it ran slower than copying each sphere.

Answer 1

No one creates this many spheres because no human mind can sufficiently process the information contained in the resulting cloud. A simplified display suffices.

To build one:

1. Evolve the locations of your spheres using whatever physics you like.
2. Build a 3D matrix with a specified cell size with each cell having an initial value of zero (you should probably implement this as a hash table to avoid having a too-large matrix).
3. Determine which cell the center of each sphere falls into and set that cell's value to the number of spheres whose centers it contains.
4. Take a threshold value and eliminate all the cells whose values fall below this threshold. Turn the rest of the cells into spheres.
5. OR Make a sphere for every cell that has a non-zero value (this assumes that the spheres tend to cluster into the cells in a way which significantly reduces their number).

This gives you O(N) time in moving from your physics to your display representation. It will not be perfect since, upon close examination, spheres may appear to pop in and out of existence. But it's a start.