If I use a standard NSWindow to host a NSOpenGLView which extends for all the window frame, the two bottom corners of the window are automatically rounded.
When swithcing to NSBorderlessWindowMask I have to handle corner rounding myself.
I have already implemented a transparent custom NSWindow and a rounded custom NSView and they both work fine.
After that I have implemented a transparent NSOpenGLContext by setting NSOpenGLCPSurfaceOpacity to 0.
If I set a color background for the OpenGL context, the view is drawn correctly and I obtain the desired rounded corner.
But, since the app is a movie player, I need to draw the texture corresponding to every movie frame.
When I do this (using glTexCoord2f and glVertex2f) the texture is drawn till corners and therefore the image is drawn till outside of the rounded corners and I loose the rounded aspect of my window.
What does the system do when the window is standard and non NSBorderlessWindowMask that I can't seem to be able to reproduce?
What is the best way to round the corner of the texture while drawing it to the frame buffer?
You could apply the texture to a geometry with rounded corners, use an additional alpha mask on the movie texture with rounded corners, or use the stencil test to round off your viewport's corners.