I'm using spherically billboarded sprites along with 3D objects. Because the quad leans backwards to match the camera angle, it intersects with 3D objects immediately behind it. It is more noticeable when the camera angle is very large.The following link provides a very clear visual. http://answers.unity3d.com/questions/582680/billboard-issue-in-front-of-3d-object.html
Is there an efficient way to resolve this?
The best solution I could come up with was to use cylindrical billboarding for depth calculations and spherical for the quad's actual position. This allows you to use spherical billboarding while ensuring the quad's depth remains constant.
For reference here are the billboarding ModelView Matrixes. [x]: implies the value is left as is.
Cylindrical mvMatrix Spherical mvMatrix
[1][x][0][x] [1][0][0][x]
[0][x][0][x] [0][1][0][x]
[0][x][1][x] [0][0][1][x]
[x][x][x][x] [x][x][x][x]
First modify the ModelViewMatrix for cylindrical billboarding and generate a depth vertex as such:
depthV = projectionMatrix * (mvm * vertex);
Next set the second column values for spherical billboarding and create the quad as usual:
mvm[1][0] = 0; mvm[1][2] = 0; mvm[1][1] = 1;
gl_Position = projectionMatrix * (mvm * vertex);
Finally send depthV to the fragment shader and use it for the depth calculation.
float ndcDepth = depthV.z / depthV.w;
gl_FragDepth = ((gl_DepthRange.diff * ndcDepth ) + gl_DepthRange.near + gl_DepthRange.far) / 2.0;
Scaling should be done before applying the ModelView Matrixes.