I'm building a recorder using a BlackMagic DeckLink Card.
The program is using OpenGL to display the Frames and FFMPEG (with the glReadPixels() method) to record them.
I'm setting the viewport in my program to apply a automatic letter-/pillarbox whether the Image/Monitor is 16:9/4:3.
The problem with that ist that when i capture these frames of this viewport they get of course recorded in the resolution my viewport is (i.e. Full Hd source -> viewport due to monitor with 1600x1200 resolution -> letterbox down to 1600x900) and so FFMPEG records those 1600x1200 with black bars at the top/bottom.
Is there any possibility to grab the RAW Frame before it gets passed trough the setViewport function and all the rescaling stuff?
Well, at some point the image is passed to OpenGL. Why don't you just take that data and pass it to FFMPEG directly instead of doing the lengthy, inefficient and expensive round trip through OpenGL.
If OpenGL is used for realtime colorspace conversion, that I suggest you do that rendering to a FBO with a texture attached with the size of the video resolution, use a glReadPixels on that FBO (preferrably into a PBO); and finally draw that rendered to texture onto the main screen in that window resolution.
However if you can simply feed the raw frames directly to FFMPEG (which can do colorspace conversions as well) I strongly suggest you do that.