I'm using glut for a game right now and I'm trying to keep the mouse inside the window. This isn't a first person shooter so locking it in the center is no good. I already know about glutWarpPointer(int, int); and I've trying things that work (kinda).
I've tried having the mouse warp back to the nearest edge of the window when it leaves, this works, but for a split second you see the mouse outside of the window and teleport back in. I don't want that, I want it to seem like the mouse just hits the edge of the window and stops going any further in that direction, while keeping movement in any other available direction. Like you would expect it to work.
This is not exactly an answer to your question, but it is an answer to your problem!
Almost every game has its own cursors. They would hide the mouse, and draw the cursor manually where the mouse should be positioned.
If you get your own cursor image and do as I said, you can simply draw the curser at the edge of the screen, even though the mouse position reads out of boundaries. Then you can warp the mouse back in.