I'm having a problem with color picking and antialiasing in OpenGL. When AA is activated results from glReadPixels are obviously wrong on object edges and object intersections. For example:
I render a box #28 (RGBA: 28, 0, 0, 0) near a box #32 (RGBA: 32, 0, 0, 0). With AA, I can get a wrong ReadPixel value (e.g. 30) where the cube and triangle overlap, or value of 14 on boxes edge, due to the AA algorithm.
I have ~4000 thousand objects I need to be able to pick (it's a jigsaw puzzle game). It is vital to be able to select objects by shape.
I've tried to disable AA with glDisable(GL_MULTISAMPLE) but it does not works with certain AA modes (I read it depends on AA implementation - SS, MS, CS ..)
So, how do I pick an underlying object?
Why not use an FBO as your pick buffer?