I have a large point cloud and each point has {x,y,z,r,g,b}. I am able to view the point cloud from a given camera pose and looking angle, and save that into a screenshot. I need to make changes to my point cloud based on this image.
How do I get the original coordinates of each pixel in the projected image?
Basically, instead of just projecting each point on the pixel, I also want to store each pixel's original location (or any other meta-data).
I want to save N*M*3 (RGB) + N*M*3 (coordinates) instead of just N*M*3 matrix for each view. Note that I need this information every time and computational overhead is not an issue.
Without some more details of your setup or requirements, I'll assume you're in a modern OpenGL (3.x+) environment. In this case, you could simply use an additional render target, and store the additional data in the other (non-color) target.
I would assume that your point cloud data is being passed to a vertex (or geometry shader). You would need to pass the original x,y,z values to the fragment shader with a varying. Add an additional out parameter to your fragment shader, and write to the original coordinates to the new out parameter.
You would then bind a target to your new output by using glBindFragDataLocation. Presumably you are already doing this with your color output. Your additional target texture would likely need to be a floating-point texture (GL_RGB32F if your driver supports it). A 'normal' color target (GL_RGBA8) would not have enough resolution.
Taking a "screenshot", you would save both your color output, and the additional target. Every pixel in the color output would correspond to the same pixel in the additional target, for which you can read the coordinates of the original point that you output. If you were going to do further processing with the GPU, you could sample both textures at the same pixel for correspondence. You would need to decide how to implement 'empty' pixels (eg. those with no object present).
You could duplicate this for any additional meta-data you require, addition and binding additional targets (presuming your setup supports enough of them). The maximum number of outputs you can have is defined by GL_MAX_DRAW_BUFFERS, but is guaranteed to be at least 8.