Unreal Engine: accessing/saving in-game images to disk without blocking game thread

Question

I am working on an Unreal based open-source UAV simulation (Microsoft AirSim) where I am trying to capture and save images from a camera that's attached to the drone. The image underneath gives an idea of how the game looks like. The rightmost view on the bottom is the actual view from the camera, the other two are just processed versions of the same image.

Right now the way it's set up in this way: There's a camera asset, which is read through the code as a capture component. The three views in the screenshot are linked to this capture component. The views are streamed without any problem as the drone is flown around in-game. But when it comes to recording screenshots, the current code sets up a TextureRenderTargetResource from this capture component, and subsequently calls ReadPixels and saves that data as an image (please see below for code flow). Using ReadPixels() as it is, is blocking the game thread directly and slowing down the whole game by a lot: drops from ~120 FPS to less than 10 FPS when I start recording.

bool saveImage() {
  USceneCaptureComponent2D* capture = getCaptureComponent(camera_type, true);
  FTextureRenderTargetResource* RenderResource = capture->TextureTarget->GameThread_GetRenderTargetResource();
  width = capture->TextureTarget->GetSurfaceWidth();
  height = capture->TextureTarget->GetSurfaceHeight();

  TArray<FColor> imageColor;
  imageColor.AddUninitialized(width * height);
  RenderResource->ReadPixels(bmp);
}

Looking at this article, it seems evident that ReadPixels() "will block the game thread until the rendering thread has caught up". The article contains sample code for a 'non-blocking' method of reading pixels (through removal of FlushRenderingCommands() and using a RenderCommandFence flag to determine when the task is done), but it doesn't significantly improve the performance: the rate at which images are saved is slightly higher, but the game thread still runs at about only 20 FPS, thus making it really hard to control the UAV. Are there any more efficient asynchronous methods that can achieve what I am trying to do, perhaps, say, in a separate thread? I am also a little confused as to why the code has no trouble streaming those images on-screen as fast as possible, but saving the images seems way more complicated. It's fine even if the images are saved to disk only at 15 Hz or so, as long as it doesn't interfere with the game's native FPS too much.

Answer 1

You can move the save-to-disk operation from Game thread to other thread.
Please check Async.h for detail

The limitation for thread is that you could not modify/add/delete uobject/uactor in other thread. Multi-thread for UE4