Desktop duplication: Rotating the frames to support portrait mode

Question

I'm using d3d11 (desktop duplication) to capture the screens and send them over the network on windows 8 and above Operating Systems. The problem is that the frames are flipped/rotated if the monitor is set in portrait mode and couldn't render properly.

After analyzing I came to know that I have had to handle the following rotation modes but I did get only limited resources related to this,

typedef enum DXGI_MODE_ROTATION { 
  DXGI_MODE_ROTATION_UNSPECIFIED  = 0,
  DXGI_MODE_ROTATION_IDENTITY     = 1,
  DXGI_MODE_ROTATION_ROTATE90     = 2,
  DXGI_MODE_ROTATION_ROTATE180    = 3,
  DXGI_MODE_ROTATION_ROTATE270    = 4
} DXGI_MODE_ROTATION;

After spending a lot of time going through the resources I had come across a code in webrtc that rotates the captured buffer using libyuv.

Here is the code I got from Webrtc for rotating the captured image buffer:

Reference : desktop_frame_rotation.cc

void RotateDesktopFrame(const DesktopFrame& source,
                        const DesktopRect& source_rect,
                        const Rotation& rotation,
                        const DesktopVector& target_offset,
                        DesktopFrame* target) {
  RTC_DCHECK(target);
  RTC_DCHECK(DesktopRect::MakeSize(source.size()).ContainsRect(source_rect));
  // The rectangle in |target|.
  const DesktopRect target_rect =
      RotateAndOffsetRect(source_rect, source.size(), rotation, target_offset);
  RTC_DCHECK(DesktopRect::MakeSize(target->size()).ContainsRect(target_rect));

  if (target_rect.is_empty()) {
    return;
  }

  int result = libyuv::ARGBRotate(
       source.GetFrameDataAtPos(source_rect.top_left()), source.stride(),
       target->GetFrameDataAtPos(target_rect.top_left()), target->stride(),
       source_rect.width(), source_rect.height(),
       ToLibyuvRotationMode(rotation));
  RTC_DCHECK_EQ(result, 0);
}

} 

It's fine but libYuv doesn't have GPU support and it would be really slow to rotate the screens using CPU power.

Also I got a stackoverflow thread that discusses about frame rotations through directX itself but that too incomplete.

It would be really appreciable if someone could help me with this problem.

Answer 1

Microsoft Media Foundation Video Processor MFT can help with rotation while doing the work on GPU whenever possible. See IMFVideoProcessorControl::SetRotation method description specifically.

If Media Foundation sounds complicated (which it indeed is) but you are familiar with Direct3D 11 in general, you can use what Direct3D 11 instead: either by utilizing rotation video processor or rendering the texture with respectively rotated quad.