I'm moving my renderer from render passes to dynamic rendering and am seeing validation messages on Linux (Nvidia 550.78 to be specific) that I'm not seeing on Windows (AMD 24.3.1 to be specific). Here are a few lines that I've formatted for legibility:
[17:01:00.273][8314]: Swapchain image 0xcad092000000000d: 'ColorAttachmentOptimal'->'PresentSrcKHR'
[17:01:00.273][8314]:
Validation Error: [ SYNC-HAZARD-WRITE-AFTER-READ ]
Object 0: handle = 0x555555b41130, type = VK_OBJECT_TYPE_QUEUE; |
MessageID = 0x376bc9df |
vkQueueSubmit():
Hazard WRITE_AFTER_READ for entry 0,
VkCommandBuffer 0x5555582fad00[],
Submitted access info (submitted_usage: SYNC_IMAGE_LAYOUT_TRANSITION,
command: vkCmdPipelineBarrier,
seq_no: 1,
VkImage 0xcad092000000000d[],
reset_no: 30).
Access info (prior_usage: SYNC_PRESENT_ENGINE_SYNCVAL_PRESENT_ACQUIRE_READ_SYNCVAL,
read_barriers: VK_PIPELINE_STAGE_2_COLOR_ATTACHMENT_OUTPUT_BIT|VK_PIPELINE_STAGE_2_BOTTOM_OF_PIPE_BIT,
,
batch_tag: 677,
vkAcquireNextImageKHR aquire_tag:677
: VkSwapchainKHR 0xe88693000000000c[],
image_index: 0
image: VkImage 0xcad092000000000d[]).
[17:01:00.273][8314]:
Validation Error: [ SYNC-HAZARD-WRITE-AFTER-WRITE ]
Object 0: handle = 0x555555b41130, type = VK_OBJECT_TYPE_QUEUE; |
MessageID = 0x5c0ec5d6 |
vkQueueSubmit():
Hazard WRITE_AFTER_WRITE for entry 0,
VkCommandBuffer 0x5555582fad00[],
Submitted access info (submitted_usage: SYNC_IMAGE_LAYOUT_TRANSITION,
command: vkCmdPipelineBarrier,
seq_no: 2,
VkImage 0x2e2941000000001f[],
reset_no: 30).
Access info (prior_usage: SYNC_LATE_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_WRITE,
write_barriers: 0,
queue: VkQueue 0x555555b41130[],
submit: 88,
batch: 0,
batch_tag: 663,
command: vkCmdEndRenderingKHR,
command_buffer: VkCommandBuffer 0x555558313000[],
seq_no: 11,
reset_no: 28).
[17:01:00.289][8314]: Swapchain image 0x967dd1000000000e: 'Undefined'->'ColorAttachmentOptimal'
[17:01:00.289][8314]: Image 0x2e2941000000001f: 'Undefined'->'DepthAttachmentOptimal'
The first and last two lines are my own logging, while the "middle two lines" are broken out validation messages. It looks like there's some formatting nonsense going on, but I preserved the missing information and was as consistent as I could be with my formatting.
Anyway, after I submit the command buffer that has the transition pipeline barrier in it, I get a write-after-read error. Here's the code that actually controls when the barriers are created. Just before I call beginRenderingKHR(), I transition the swapchain image and depth buffers:
auto &color_buffer = *Renderer::swapchain().images()[Renderer::image_index()];
color_buffer.transition_layout(Renderer::cmd_buffer(),
vk::ImageLayout::eUndefined,
vk::ImageLayout::eColorAttachmentOptimal);
_depth_buffer.transition_layout(Renderer::cmd_buffer(),
vk::ImageLayout::eUndefined,
vk::ImageLayout::eDepthAttachmentOptimal);
Then, just after calling endRenderingKHR() I transition the color buffer:
auto &color_buffer = *Renderer::swapchain().images()[Renderer::image_index()];
color_buffer.transition_layout(Renderer::cmd_buffer(),
vk::ImageLayout::eColorAttachmentOptimal,
vk::ImageLayout::ePresentSrcKHR);
And here's the image transition code (vkSwapchainImage doesn't apply to the depth buffer, but the logic is the same):
void vkSwapchainImage::transition_layout(vkCmdBuffer const &cmd_buffer,
vk::ImageLayout const old_layout,
vk::ImageLayout const new_layout)
{
BTX_TRACE("Swapchain image {}: '{:s}'->'{:s}'",
_handle,
vk::to_string(old_layout),
vk::to_string(new_layout));
vk::ImageMemoryBarrier barrier {
.srcAccessMask = { },
.dstAccessMask = { },
.oldLayout = old_layout,
.newLayout = new_layout,
.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.image = _handle,
.subresourceRange {
.aspectMask = vk::ImageAspectFlagBits::eColor,
.baseMipLevel = 0u,
.levelCount = 1u,
.baseArrayLayer = 0u,
.layerCount = 1u,
}
};
vk::PipelineStageFlags src_stage = vk::PipelineStageFlagBits::eNone;
vk::PipelineStageFlags dst_stage = vk::PipelineStageFlagBits::eNone;
if(old_layout == vk::ImageLayout::eUndefined) {
barrier.srcAccessMask = vk::AccessFlagBits::eNone;
if(new_layout == vk::ImageLayout::eColorAttachmentOptimal) {
barrier.dstAccessMask = vk::AccessFlagBits::eColorAttachmentRead
| vk::AccessFlagBits::eColorAttachmentWrite;
src_stage = vk::PipelineStageFlagBits::eTopOfPipe;
dst_stage = vk::PipelineStageFlagBits::eColorAttachmentOutput;
}
else if(new_layout == vk::ImageLayout::eDepthAttachmentOptimal) {
barrier.dstAccessMask =
vk::AccessFlagBits::eDepthStencilAttachmentRead
| vk::AccessFlagBits::eDepthStencilAttachmentWrite;
src_stage = vk::PipelineStageFlagBits::eTopOfPipe;
dst_stage = vk::PipelineStageFlagBits::eEarlyFragmentTests
| vk::PipelineStageFlagBits::eLateFragmentTests;
}
else {
BTX_CRITICAL("Unsupported image layout transition");
return;
}
}
else if(old_layout == vk::ImageLayout::eColorAttachmentOptimal) {
barrier.srcAccessMask = vk::AccessFlagBits::eColorAttachmentRead
| vk::AccessFlagBits::eColorAttachmentWrite;
barrier.dstAccessMask = vk::AccessFlagBits::eNone;
src_stage = vk::PipelineStageFlagBits::eColorAttachmentOutput;
dst_stage = vk::PipelineStageFlagBits::eBottomOfPipe;
}
else {
BTX_CRITICAL("Unsupported image layout transition");
return;
}
cmd_buffer.native().pipelineBarrier(
src_stage, // Source stage
dst_stage, // Destination stage
{ }, // Dependency flags
nullptr, // Memory barriers
nullptr, // Buffer memory barriers
{ barrier } // Image memory barriers
);
_layout = barrier.newLayout;
}
I made an effort to copy the image transitions exactly from the official dynamic rendering example code, and they seem fine on Windows/AMD, but of course I'd like to understand either what's incomplete or what's wrong.
And of course any structural improvements for handling image transitions better would be very welcome. =)
The real "answer" to this question is that I still need to study synchronization more. No surprise there.
However, I was able to knock out the sync validation errors I was seeing by moving my existing code to use the sync2 extension. I used vkguide.dev's new setup as a framework and went from there. Now, the chain of events looks like the following.
I begin a one-time-submit command buffer recording. Next, I transition the color buffer and depth buffer and begin rendering.
void vkColorDepth::begin() {
auto &color_buffer = *Renderer::swapchain().images()[Renderer::image_index()];
color_buffer.transition_layout(Renderer::cmd_buffer(),
vk::ImageLayout::eUndefined,
vk::ImageLayout::eColorAttachmentOptimal);
_depth_buffer.transition_layout(Renderer::cmd_buffer(),
vk::ImageLayout::eUndefined,
vk::ImageLayout::eDepthAttachmentOptimal);
vk::Rect2D const render_area = {
.offset { .x = 0u, .y = 0u },
.extent {
.width = Renderer::swapchain().size().width,
.height = Renderer::swapchain().size().height,
},
};
Renderer::cmd_buffer().begin_rendering(
vk::RenderingInfoKHR {
.pNext = nullptr,
.flags = { },
.renderArea = render_area,
.layerCount = 1u,
.viewMask = 0u,
.colorAttachmentCount = 1u,
.pColorAttachments = &_color_attachments[Renderer::image_index()],
.pDepthAttachment = &_depth_attachment,
.pStencilAttachment = nullptr,
}
);
}
Commands get recorded, then I transition the color buffer again:
void vkColorDepth::end() {
Renderer::cmd_buffer().end_rendering();
auto &color_buffer = *Renderer::swapchain().images()[Renderer::image_index()];
color_buffer.transition_layout(Renderer::cmd_buffer(),
vk::ImageLayout::eColorAttachmentOptimal,
vk::ImageLayout::ePresentSrcKHR);
}
And finally the commands get submitted:
void Renderer::_submit_commands() {
auto const &frame_sync = _frame_sync[_image_index];
auto const cmd_submit_info = vk::CommandBufferSubmitInfo {
.pNext = nullptr,
.commandBuffer = frame_sync.cmd_buffer().native(),
.deviceMask = { },
};
auto const wait_info = vk::SemaphoreSubmitInfoKHR {
.pNext = nullptr,
.semaphore = frame_sync.present_semaphore(),
.value = { },
.stageMask = vk::PipelineStageFlagBits2KHR::eColorAttachmentOutput,
.deviceIndex = { },
};
auto const signal_info = vk::SemaphoreSubmitInfoKHR {
.pNext = nullptr,
.semaphore = frame_sync.queue_semaphore(),
.value = { },
.stageMask = vk::PipelineStageFlagBits2KHR::eAllGraphics,
.deviceIndex = { },
};
auto const queue_submit_info = vk::SubmitInfo2KHR {
.pNext = nullptr,
.flags = { },
.waitSemaphoreInfoCount = 1u,
.pWaitSemaphoreInfos = &wait_info,
.commandBufferInfoCount = 1u,
.pCommandBufferInfos = &cmd_submit_info,
.signalSemaphoreInfoCount = 1u,
.pSignalSemaphoreInfos = &signal_info,
};
auto const result = _device.graphics_queue().native().submit2KHR(
1u,
&queue_submit_info,
frame_sync.queue_fence()
);
if(result != vk::Result::eSuccess) {
BTX_CRITICAL("Failed to submit commands to device queue: '{}'",
vk::to_string(result));
}
}
Now, here's what the swapchain image transition code looks like:
void vkSwapchainImage::transition_layout(vkCmdBuffer const &cmd_buffer,
vk::ImageLayout const old_layout,
vk::ImageLayout const new_layout)
{
// BTX_TRACE("Swapchain image {}: '{:s}'->'{:s}'",
// _handle,
// vk::to_string(old_layout),
// vk::to_string(new_layout));
vk::ImageMemoryBarrier2KHR barrier {
.pNext = nullptr,
.srcStageMask = vk::PipelineStageFlagBits2KHR::eAllCommands,
.srcAccessMask = vk::AccessFlagBits2KHR::eMemoryWrite,
.dstStageMask = vk::PipelineStageFlagBits2KHR::eAllCommands,
.dstAccessMask = vk::AccessFlagBits2KHR::eMemoryRead
| vk::AccessFlagBits2KHR::eMemoryWrite,
.oldLayout = old_layout,
.newLayout = new_layout,
.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
.image = _handle,
.subresourceRange {
.aspectMask = vk::ImageAspectFlagBits::eColor,
.baseMipLevel = 0u,
.levelCount = VK_REMAINING_MIP_LEVELS,
.baseArrayLayer = 0u,
.layerCount = VK_REMAINING_ARRAY_LAYERS,
}
};
if(old_layout == vk::ImageLayout::eUndefined) {
if(new_layout == vk::ImageLayout::eColorAttachmentOptimal) {
barrier.srcAccessMask = vk::AccessFlagBits2KHR::eColorAttachmentRead
| vk::AccessFlagBits2KHR::eColorAttachmentWrite;
barrier.dstAccessMask = vk::AccessFlagBits2KHR::eColorAttachmentRead
| vk::AccessFlagBits2KHR::eColorAttachmentWrite;
}
else {
BTX_CRITICAL("Image {}: unsupported swapchain image layout "
"transition: '{:s}'->'{:s}'",
_handle,
vk::to_string(old_layout),
vk::to_string(new_layout));
return;
}
}
else if(old_layout == vk::ImageLayout::eColorAttachmentOptimal) {
if(new_layout == vk::ImageLayout::ePresentSrcKHR) {
barrier.srcAccessMask = vk::AccessFlagBits2KHR::eColorAttachmentRead
| vk::AccessFlagBits2KHR::eColorAttachmentWrite;
barrier.dstAccessMask = { };
}
else {
BTX_CRITICAL("Image {}: unsupported swapchain image layout "
"transition: '{:s}'->'{:s}'",
_handle,
vk::to_string(old_layout),
vk::to_string(new_layout));
return;
}
}
else {
BTX_CRITICAL("Image {}: unsupported swapchain image layout transition: "
"'{:s}'->'{:s}'",
_handle,
vk::to_string(old_layout),
vk::to_string(new_layout));
return;
}
auto dep_info = vk::DependencyInfoKHR {
.pNext = nullptr,
.dependencyFlags = { },
.memoryBarrierCount = { },
.pMemoryBarriers = { },
.bufferMemoryBarrierCount = { },
.pBufferMemoryBarriers = { },
.imageMemoryBarrierCount = 1u,
.pImageMemoryBarriers = &barrier,
};
cmd_buffer.native().pipelineBarrier2KHR(dep_info);
}
I'll skip posting the depth buffer transition code, because structurally it's basically the same.
On surprisingly useful thing about this exercise was learning that sync2's validation warnings are a lot more informative than sync1's were. Other than outright ignoring the warning for using the ALL_COMMANDS stage mask, I was actually able to go from one warning to the next after the initial setup of using vkguide.dev's sledgehammer approach to layout transition sync.
I still have to actually work through it to garner some understanding, rather than the copy/paste/follow-the-prompts I've done thus far, but it's nice to have a validated stepping stone for that.