glModelMatrix function is used to modify either projection or model matrices and compute vertices positions properly. However, since i use VAO, VBO and shader program the magic doesn't happens anymore. (same issues with gluPerspective or gluOrtho)
Should i compute model and projection matrices on my own and send them to shader ?
Yes, the general idea is to perform matrix computations on your own. Actually no serious 3D program ever made use of the old OpenGL matrix stack. There's no benefit in using the old OpenGL matrix functions, except that they make things a little bit more convenient for beginners (no, there's also no GPU acceleration for that). Quite frankly, OpenGL should never have had them in the first place, and GLU should have provided the basic matrix functions.
Any matrix math library that can work with 4×4 matrices will do. GLM is very popular among the c++ folks, and there's a Java conversion called jGLM at https://github.com/jroyalty/jglm