How to create a letterbox shader to reserve an aspect ratio in wgpu

Question

I want a shader that reserves the aspect ratio like you see in many games. I have a quad that covers the screen in a second render pass along with the frame as a texture, screen width and height, and target resolution width and height. However I cant figure out the math.

Here is the data I have to work with:

struct PostprocessOutput {
    @builtin(position) clip_position: vec4<f32>,
    @location(0) tex_coords: vec2<f32>,
    @location(1) resolution: vec2<f32>,
    @location(2) window_size: vec2<f32>,
}

Answer 1

The issue of filling the viewport with a full-screen quad is related to the model view projection matrix you are using. Perhaps this problem can be transformed into: How to project a standard rectangle with model coordinates of [-1, 1] to exactly fill the entire viewport.

pub fn cal_fullscreen_mvp(viewport: Size<f32>) -> glam::Mat4 {
    let fovy: f32 = 75.0 / 180.0 * std::f32::consts::PI;
    let factor = fullscreen_factor(viewport, fovy);

    let p_matrix = Mat4::perspective_rh(fovy, viewport.width / viewport.height, 0.1, 100.0);
    let vm_matrix = Mat4::from_translation(glam::vec3(0.0, 0.0, factor.0));
    let scale_matrix = Mat4::from_scale(glam::Vec3::new(factor.1, factor.2, 1.0));

    p_matrix * vm_matrix * scale_matrix
}
  

pub fn fullscreen_factor(viewport: Size<f32>, fovy: f32) -> (f32, f32, f32) {
    let mut sx = 1.0;
    let mut sy = 1.0;
    
    let ratio = if viewport.height > viewport.width {
        let ratio = viewport.height / viewport.width;
        sy = ratio;
        ratio
    } else {
        sx = viewport.width / viewport.height;
        1.0
    };

    let translate_z = -(ratio / (fovy / 2.0).tan());
    
    (translate_z, sx, sy)
}