Weird rchit.glsl shader behaviour with the hitAttributeEXT variable

Question

I was working on some Vulkan raytracing renderers when I encountered a visual bug. When I tried to debug it I noticed it came down to my rchit.glsl (ray closest hit) code and had to do with the hitAttributeEXT variable. After breaking it down to the simplest reproducible scenario I became dumbfounded that whenever a computation is performed on said parameter I get inexplicable graphical output.

#version 460
#extension GL_EXT_ray_tracing : require
#extension GL_EXT_nonuniform_qualifier : enable
#extension GL_EXT_scalar_block_layout : enable

layout(location = 0) rayPayloadInEXT vec3 hitValue; // This is simply the resulting output

hitAttributeEXT vec2 attribs; // barycentric coordinates of the triangle hit by a ray

void main()
{
    // For testing purposes I either uncommented or commented out a separate section
    // Raytracing Shader Test 1
    // setting hit value directly as attribs (this works as expected)
    hitValue = vec3(1.0, attribs);

    // Raytracing Shader Test 2
    // Adding a branch (this does not work as expected)
    if (attribs.x > 0.5)
    {
        hitValue = vec3(1.0, 1.0, 1.0);
    } else {
        hitValue = vec3(0.0, 0.0, 0.0);
    }

    // Raytracing Shader Test 3
    // fixed value for sanity check
    hitValue = vec3(0.0, 1.0, 0.0);
}

In the below I made a couple tests with a projected ico-sphere.

Raytracing Shader Test 1

So here things appear to work as expected, triangles have red on one corner, red+blue=purple on another and red+green=yellow on the final corner.

Raytracing Shader Test 2

So this is completely perplexing to me, I can't see how the color red is even possible given my shader setup.

Raytracing Shader Test 3

Using a constant color output

Regular Rasterization

And just for some context this is how the object really looks with a rasterization renderer.

---

some additional info

Vulkan vers:

• "vulkan-headers/1.3.211.0",

• "vulkan-loader/1.3.211.0",

• "vulkan-validationlayers/1.3.211.0",

glslc from VulkanSDK-1.3.211.0:

$ glslc --version

shaderc v2022.1 v2022.1

spirv-tools v2022.2-dev v2022.1-61-geed5c76a

glslang 11.1.0-457-gabbe4664

Target: SPIR-V 1.0

glslc command:

glslc --target-env=vulkan1.3 rayhit_shader.rchit -o rayhit_shader.spv

Answer 1

I posted this on reddit and someone suggested it had something to do with my shader binding table setup and that's where I discovered the issue, in case someone else stumbles into something similar...

It had to do with a shader binding table and raytracing pipeline layout mismatch of the shader index.

I had something like the below in my raytracing pipeline layout

VkRayTracingShaderGroupCreateInfoKHR raygen{VK_STRUCTURE_TYPE_RAY_TRACING_SHADER_GROUP_CREATE_INFO_KHR};
raygen.type = VK_RAY_TRACING_SHADER_GROUP_TYPE_GENERAL_KHR;
raygen.generalShader = 0;

VkRayTracingShaderGroupCreateInfoKHR rayhit{VK_STRUCTURE_TYPE_RAY_TRACING_SHADER_GROUP_CREATE_INFO_KHR};
rayhit.type = VK_RAY_TRACING_SHADER_GROUP_TYPE_TRIANGLES_HIT_GROUP_KHR;
rayhit.closestHitShader = 1;

VkRayTracingShaderGroupCreateInfoKHR raymiss{VK_STRUCTURE_TYPE_RAY_TRACING_SHADER_GROUP_CREATE_INFO_KHR};
raymiss.type = VK_RAY_TRACING_SHADER_GROUP_TYPE_GENERAL_KHR;
raymiss.generalShader = 2

std::vector<VkRayTracingShaderGroupCreateInfoKHR> rt_shader_groups = { raygen, rayhit, raymiss };

Meanwhile I had this in my shader binding table setup

raygen_sbt_region.deviceAddress = sbt_address;
raymiss_sbt_region.deviceAddress = sbt_address + raygen_sbt_region.size;
rayhit_sbt_region.deviceAddress = sbt_address + raygen_sbt_region.size + raymiss_sbt_region.size;

Syncing the two properly fixed the issue. I have no idea how the raymiss and hit shaders worked sometimes previously.