How to use the ffmpeg overlay_cuda filter to make a SBS video?

FFMPEG a few months ago launched the new version of FFMPEG with the new filter "overlay_cuda", this filter makes the same as the "overlay" but using an Nvidia card for applying it.

I found on the FFMPEG website description of the filter, but no examples of how to use it. The only examples I found are from the developer commits but are to put a video or a photo over another video.

I before the normal overlay filter to do this using a nullsrc image with the twice of width but now I don't know how to with this filter.

Commit description: https://patchwork.ffmpeg.org/project/ffmpeg/patch/[email protected]/ ffmpeg documentation webpage: https://ffmpeg.org/ffmpeg-filters.html#overlay_005fcuda-1

I hope you can help me.

Update:

I made this FFmpeg order that:

Input each video.
The first video creates padding to the right and then is uploaded to the card's memory.
With overlay Cuda, the other video puts on the right of the original video.

ffmpeg -y -loglevel info \
-i $video_1  \
-hwaccel cuda -hwaccel_output_format cuda -i $video_2 \
-filter_complex \
" \
[0:v]pad=w=2*iw:h=ih:x=0:y=0,hwupload_cuda[base];
[base][1:v]overlay_cuda=x=800:y=0" \
-an -c:v h264_nvenc overlay_test.mp4

But I get this error message:

[overlay_cuda @ 0x55fdec4b2ec0] Can't overlay nv12 on yuv420p 
[Parsed_overlay_cuda_2 @ 0x55fdec4b2d80] Failed to configure output pad on Parsed_overlay_cuda_2
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0

I have issues with the pixel formats, I hope you can help me.

Update 2:

I finally solved the pixel format issue and now makes the overlay with the padding (adds space to the overlay video)

This is the command that succeeds:

ffmpeg -y -loglevel info \
-i $video_1  \
-hwaccel cuda -hwaccel_output_format cuda -i $video_2 \
-filter_complex \
" \
[0:v]pad=w=2*iw:h=ih:x=0:y=0,hwupload_cuda,scale_npp=format=nv12[base];
[base][1:v]overlay_cuda=x=800:y=0" \
-an -c:v h264_nvenc overlay_test.mp4

Now I want to change x=800 to a variable like x=iw+1 but it seems this filter doesn't support that. Is there a way to set a global variable?

Solution

I got it! After reading again what I can do with the Cuda filters, I found that the scale_npp not only resizes the videos, but also you can change the pixel format.

So after a few test I found a very nice solution:

ffmpeg -y -loglevel info \
-hwaccel cuda -hwaccel_output_format cuda -i $video_1  \
-hwaccel cuda -hwaccel_output_format cuda -i $video_2 \
-filter_complex \
" \
[0:v]scale_npp=640:-2:format=yuv420p,hwdownload,pad=w=2*iw:h=ih:x=0:y=0,hwupload_cuda,scale_npp=format=nv12[base];
[1:v]scale_npp=640:-2:format=nv12[overlay_video];
[base][overlay_video]overlay_cuda=x=640:y=0" \
-an -c:v h264_nvenc overlay_test.mp4

What makes this FFMPEG command:

Input two videos with Cuda decoding.
The first video [0:v]:
- Escale to 640 pixels width keeping the aspect radio with a YUV420P pixel format
- Download from the GPU memory to the system memory
- Apply a Padding filter to add 640 pixels width to the right of the video
- Upload again to the GPU memory
- Change the pixel format to nv12
- Label as [base]
The second video [1:v]
- Escale to 640 pixels width keeping the aspect radio with a NV12 pixel format
- Label as [overlay_video]
Apply overlay_cuda filter
- [base] video as the background video
- [overlay_video] video as the foreground video
- Insert [overlay_video] 640 pixels to the right as the [base] video
Finish the encoding
- -an as Audio null (This can be deleted, in the real usage you need to mix the audio signals or choose one from the two videos, or even add an external audio source.)
- -c:v h264_nvenc Encode the video using the GPU with the h264 codec (Here you can change to your needs).

The only drawback is that you need to set a resolution from beforehand, there is no way to set the input resolution (Like in the normal overlay filter). Fortunately, you can make this like a variable for a script and use ffprobe to get this variable before.