I have been trying to get a 3D scatter plot to display using InstancedMesh of three.js via a React app (therefore using react three fiber).
After a few days I was able to get the 3D scatter plot of spheres and also color them. However I want to be able to select/highlight individual spheres on mouse click/hover. I followed a simple tutorial that used onPointerOver and onPointerOut but that does not seem to work perhaps because it was not meant for InstancedMesh objects.
It looks like I need to use raycaster, but it is not clear to me how to do it. Any suggestions would be really helpful.
Steps to setup -
npx create-react-app demo
cd demo
npm install three
npm i @react-three/fiber
npm i @react-three/drei
Current code that shows the differently colored spheres -
App.jsx
import React from 'react'
import { Suspense } from "react";
import { Canvas } from "@react-three/fiber";
import { OrbitControls } from "@react-three/drei";
import Spheres from "./IScatter";
import * as THREE from "three";
function App() {
return (
<div>
<Canvas style={{width:"100%",height:"100vh"}}>
<OrbitControls enableZoom={true} />
<ambientLight intensity={0.5} />
<pointLight position={[10, 10, 10]}/>
<Suspense fallback={null}>
<primitive object={new THREE.AxesHelper(1.5)} />
<Spheres />
</Suspense>
</Canvas>
</div>
);
}
export default App;
IScatter.jsx
import * as THREE from "three";
import React, { useRef, useState } from "react";
import { useEffect } from "react";
import { DoubleSide } from "three";
const points = [ [1, 0, -1], [0, 1, -0.5], [0.5, 0.5, 0.5], [1,0.25,-1], [1,0,1], [0,1,0.5] ];
const content = ["Hello", "World", "Hello World", "Shoes", "Drone", "Foo Bar"];
const colors = [0,0,0,5,5,5];
const tempSphere = new THREE.Object3D();
const Spheres = () => {
const material = new THREE.MeshLambertMaterial({ opacity: 0.5, side: THREE.DoubleSide, transparent: true,});
const spheresGeometry = new THREE.SphereBufferGeometry(0.25, 15, 15);
const ref = useRef();
const [active, setActive] = useState(false);
const [hover, setHover] = useState(false);
useEffect(() => {
points.map(function (val, row) {
console.log(val, row);
tempSphere.position.set(val[0], val[1], val[2]);
tempSphere.updateMatrix();
ref.current.setMatrixAt(row, tempSphere.matrix);
ref.current.setColorAt(row, new THREE.Color(`hsl(${colors[row]*100}, 100%, 50%)`));
});
ref.current.instanceMatrix.needsUpdate = true;
// ref.current.instanceColor.needsUpdate = true;
});
return (
<instancedMesh onClick={() => {
setActive(!active);
}}
onPointerOver={() => {
setHover(true);
}}
onPointerOut={() => {
setHover(false);
}} ref={ref} rotation={[0,0,0]} args={[spheresGeometry, material, 15]} opacity={active?0.9:0.5} />
);
};
export default Spheres;
I would like to change the opacity of the selected sphere and perhaps make them a little larger so that we know what it selected. If possible I would also like to show the HTML content associated with a sphere, adjacent to the canvas, away from the scatterplot of spheres, whenever a sphere is selected.
EDIT: I essentially want to implement this - https://threejs.org/examples/webgl_instancing_raycast but using react three fiber and to also show associated content of the spheres.
I tried to work with this the best I could and here are the results.
Here's my sandbox: https://codesandbox.io/s/relaxed-rgb-yb1v1v?file=/src/IScatter.jsx
First of all, I made a new Scene.jsx, because Three Fiber apparently doesn't like certain (can't remember what exactly) reference or hook usage outside of Canvas.
My thought here was also that the Scene should be responsible for handling any mouse or camera events and only pass the changes one way or another to other components.
Basically App became just this:
function App() {
return (
<div>
<Canvas style={{ width: "100%", height: "100vh" }}>
<Scene />
</Canvas>
</div>
);
}
And all of your old App code now lies inside Scene.
Looking at the raycast example code, we can see that in order to use the raycaster, we need... well... the raycaster instance for one, but also the camera instance and a mouse position.
To get camera instance with Three Fiber, I found this SO question and answer, which guides us to use useThree hook:
const {
camera,
gl: { domElement }
} = useThree();
Now this doesn't help us that much by itself, because the useThree().camera is not exactly "bound" to the OrbitControls component you are using. To fix this, we use the args prop of the OrbitControls component to, afaik, bind/pass our camera to it:
<OrbitControls
enableZoom={true}
args={[camera, domElement]}
onChange={handleControls}
/>
You might notice that there is also a onChange-prop there. The event does trigger whenever camera rotation etc. changes, but for some unknown reason, the event object does not contain basically any other information at all other than "something has changed".
I was hoping that the event object would include camera information or something. Either way, because it seems to work like that and since we've passed the camera to the OrbitControls component anyway, whenever this change event triggers, we know that the camera has updated as well. My first instinct was to of course have something like this:
const [myCamera, setMyCamera] = useState(camera);
const handleControls = (e) => {
setMyCamera(camera);
};
So my own camera state, which just set whatever camera has whenever something has changed with the OrbitControls. Well the problem here is that while this kind of works, something like useEffect(() => { }, [myCamera]); won't ever trigger for yet another unknown reason.
So what does a frustrated React developer do when it has useState + useEffect triggering problems? Makes a "force state change" implementation and here is what the handleControls currently looks like:
const [cameraSignal, setCameraSignal] = useState(0);
const handleControls = (e) => {
setCameraSignal((s) => s + 1);
};
And then we just pass that cameraSignal to your Spheres component:
Again, the useThree().camera does update beautifully, we just didn't have a way to react to that, because useEffect won't get triggered by direct changes to it, but with this hacky cameraSignal implementation, we now have a way to trigger useEffects when camera changes.
So now we have the camera solved. Next up is mouse. This one is fortunately a lot simpler issue:
const [mousePos, setMousePos] = useState({ x: 0, y: 0 });
const handleMouse = (event) => {
const mouse = {};
mouse.x = (event.clientX / window.innerWidth) * 2 - 1;
mouse.y = -(event.clientY / window.innerHeight) * 2 + 1;
setMousePos(mouse);
};
useEffect(() => {
document.addEventListener("mousemove", handleMouse);
return () => {
document.removeEventListener("mousemove", handleMouse);
};
}, []);
That's camera and mouse solved. We can get the raycaster instance with the same useThree hook, so lets get to the actual raycasting implementation:
const ts = useThree();
const [camera, setCamera] = useState(ts.camera);
const instanceColors = useRef(new Array(points.length).fill(red));
useEffect(() => {
let mouse = new THREE.Vector2(mousePos.x, mousePos.y);
ts.raycaster.setFromCamera(mouse, camera);
const intersection = ts.raycaster.intersectObject(ref.current);
if (intersection.length > 0) {
const instanceId = intersection[0].instanceId;
ref.current.getColorAt(instanceId, color);
if (color.equals(red)) {
instanceColors.current[instanceId] = new THREE.Color().setHex(Math.random() * 0xffffff);
ref.current.instanceColor.needsUpdate = true;
}
}
}, [mousePos]);
useEffect(() => {
setCamera(ts.camera);
}, [cameraSignal]);
Here we just basically follow the raycast example code with our brand new logic to get mouse and camera data.
In places where the example uses mesh like:
const intersection = raycaster.intersectObject(mesh);
We use ref.current instead, as it seems to work like that with React Fiber and ref.current is indeed the actual InstancedMesh that you have created.
But there are some problems
I think that there are some issues regarding how to use React Fiber in general. Looking at (the documentation here)[https://openbase.com/js/@andrewray/react-three-fiber/documentation] and (here)[https://docs.pmnd.rs/react-three-fiber/api/objects], there are a lot of improvements that can be made, including to my code, which is kind of a hack to make the current solution work.
I think one of the biggest issues issues is this:
useEffect(() => {
points.map(function (val, row) {
tempSphere.position.set(val[0], val[1], val[2]);
tempSphere.updateMatrix();
ref.current.setMatrixAt(row, tempSphere.matrix);
ref.current.setColorAt(row, instanceColors.current[row]);
});
ref.current.instanceMatrix.needsUpdate = true;
//ref.current.instanceColor.needsUpdate = true;
});
This is weird because of how React Fiber components work. I think your intention here is probably to initalize the mesh, but this acts basically as a render update loop.
I had to make another ref like this to hold the changed colors in, or otherwise this "render update loop" would overwrite the colors with the initial colors all of the time:
const instanceColors = useRef(new Array(points.length).fill(red));
Then with the raycasting, I couldn't directly use setColorAt (because the above useEffect would just overwrite the color), but instead I updated the colors ref:
instanceColors.current[instanceId] = new THREE.Color().setHex(Math.random() * 0xffffff);
This all results in a working raycasting and color changing with the InstancedMesh, but I would consider some refactorings if you want to get more performance etc. out of this.
*Extra notes
I followed a simple tutorial that used onPointerOver and onPointerOut but that does not seem to work perhaps because it was not meant for InstancedMesh objects.
I believe you are right, InstancedMesh is a different beast in a sense that it is not exactly an unambiguous single 3D object you can click or hover on, quite the opposite, really. It could be that in order to use InstancedMesh, one needs to do all the initializing, updating, as well as the camera- and pointer-event stuff by hand, but I can't say for sure as this is the first time ever for me that using Three Fiber. 😅 It looks a bit messy, but I have seen something like this before, so it is not necessarily the wrongest way possible to do it like this.
Edit
I think I fixed the aforementioned issue, see IScatter.jsx of this sandbox: https://codesandbox.io/s/stoic-hertz-qbbkpu?file=/src/IScatter.jsx (It runs a lot better)
I moved some stuff out of the useEffect and even out of the component. Now it should actually initialize the InstancedMesh only once. We still have one useEffect for initialization, but it is only run once on component mount:
useEffect(() => {
let i = 0;
const offset = (amount - 1) / 2;
for (let x = 0; x < amount; x++) {
for (let y = 0; y < amount; y++) {
for (let z = 0; z < amount; z++) {
matrix.setPosition(offset - x, offset - y, offset - z);
meshRef.current.setMatrixAt(i, matrix);
meshRef.current.setColorAt(i, white);
i++;
}
}
}
}, []);
This is evident by looking at the raycast function:
meshRef.current.setColorAt(
instanceId,
new THREE.Color().setHex(Math.random() * 0xffffff)
);
We can now directly use setColorAt, without it getting overwritten on every render loop.
Furthermore I would look here and see if the mouse- and camera-events can be handled a bit better.