Is is possible to write a shader in GLSL so that I can turn OpenGL into a ray tracer? Something like the following.
glUseProgram(rayTracer);
//than do anything you do in OpenGL normally
glEnable(GL_LIGHT0);
gl....//Set light parameters
...
//Draw The theme
glDrawArray(.......);
Than the result result will rendered by raytracing. eg: physically correct illumination
I have tried this. But I failed that I can't get all the triangles vertex and color/texture in vertex/fragment shader in order to do raytracing.
Is it possible to create this kind of shader? If it's possible, how and where should I start?
No, this is not possible. Raytracing is a golbal illumination model. This means that access to the whole scene with all objects and light sources is required, for obvious reasons. The rendering pipleine OpenGL implements never sees more than a single primitive at a time. While one can do raytricing on the GPU, this is completely different from just writing some shader that can work as a drop-in replacement while keeping the rest of the rendering algorithm the same. You need to completely reogranize the data, and the GL drawing functions are of no good use at all anymore.