I'm using Unreal as a server for simulation. Now I can connect to this from Python client and transmit/receive data via UDP. But I still haven't found a proper way to capture frames and stream it to Python client to process in OpenCV.
I tried to make screenshots and save it to file, but this way is too slow.
I also tried to render to texture target, but it also is not optimal way for me, because I need render 2 times (both for camera and 2d capturer).
Now I'm find a way to access a Vulkan framebuffer (I'm on Linux), but I can't find any docs or examples, how to do it.
How to get raw framebuffer byte array in custom c++ actor?
Hello you can actually access this information from a delegate: FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent() this will be call when the frame is ready and it will return a pointer of the SWindow rendered and a FTexture2DRHIRef containing the actual backbuffer, having the RHI texture is your opportunity to captures the bytes.
UE has a class already that is doing what you want inside their implementation for the virtual camera they use with iPad. It is called FFrameGrabber, you can use that one or if you need something more sophisticated you can see how the implemented the OnBackBufferReadyToPresent delegate